Nov 01 14:14:17.912977 ip-10-0-135-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 01 14:14:17.912988 ip-10-0-135-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 01 14:14:17.912995 ip-10-0-135-142 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 01 14:14:17.913244 ip-10-0-135-142 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 01 14:14:28.038900 ip-10-0-135-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 01 14:14:28.038917 ip-10-0-135-142 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 57b276085a5a41d1944c8b8f9696027c --
Nov 01 14:16:26.439717 ip-10-0-135-142 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 14:16:26.875624 ip-10-0-135-142 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:26.875624 ip-10-0-135-142 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 14:16:26.875624 ip-10-0-135-142 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:26.875624 ip-10-0-135-142 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:26.883738 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883639    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:26.883738 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883729    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:26.883738 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883733    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:26.883738 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883736    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:26.883738 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883739    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:26.883738 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883742    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:26.883738 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883745    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:26.883738 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883748    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883751    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883754    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883757    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883760    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883763    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883766    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883769    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883771    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883774    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883777    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883780    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883782    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883785    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883788    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883791    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883793    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883797    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883800    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:26.884024 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883803    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883806    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883808    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883811    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883814    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883816    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883819    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883822    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883824    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883827    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883830    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883832    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883835    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883838    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883840    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883844    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883847    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883851    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883853    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883856    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:26.884491 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883859    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883861    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883864    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883867    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883871    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883873    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883876    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883879    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883882    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883884    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883887    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883890    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883893    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883896    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883898    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883901    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883904    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883907    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883911    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883913    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:26.884990 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883916    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883919    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883922    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883924    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883927    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883930    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883932    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883936    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883938    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883941    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883944    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883948    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883953    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883956    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883959    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883962    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883965    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883968    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883970    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:26.885471 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883973    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883976    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883978    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883981    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883984    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883987    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883990    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883992    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883995    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.883997    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884000    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884002    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884005    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884008    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884010    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884013    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884016    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884021    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884024    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884027    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:26.885936 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884031    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884850    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884858    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884861    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884864    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884867    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884870    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884873    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884876    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884879    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884882    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884885    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884888    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884891    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884893    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884896    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884899    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884902    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884905    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884907    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:26.886409 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884910    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884913    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884915    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884918    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884922    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884926    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884930    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884933    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884935    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884938    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884941    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884943    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884947    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884949    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884952    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884955    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884957    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884960    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884963    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:26.886884 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884965    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884968    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884971    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884973    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884976    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884979    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884982    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884984    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884987    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884990    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884992    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884995    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.884998    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885001    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885004    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885006    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885009    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885012    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885017    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:26.887337 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885021    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885024    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885027    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885030    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885033    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885036    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885039    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885041    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885044    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885046    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885049    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885052    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885054    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885057    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885060    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885062    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885065    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885068    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885070    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885073    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:26.887795 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885076    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885078    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885081    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885083    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885086    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885088    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885091    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885094    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885096    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885099    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885102    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885104    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885107    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885110    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885113    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885116    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885118    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885121    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885124    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885127    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:26.888261 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885130    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885132    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885135    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885137    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885140    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885143    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885146    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885149    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885151    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885673    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885680    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885683    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885686    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885689    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885692    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885695    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885698    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885701    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885703    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885706    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:26.888751 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885708    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885711    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885715    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885717    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885720    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885723    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885726    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885729    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885732    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885734    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885737    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885739    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885742    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885745    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885747    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885750    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885752    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885755    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885758    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885761    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:26.889237 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885763    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885766    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885768    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885771    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885774    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885777    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885779    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885782    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885785    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885787    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885790    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885793    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885795    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885798    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885801    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885804    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885807    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885810    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885812    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:26.889717 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885815    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885818    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885820    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885823    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885825    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885828    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885831    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885840    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885844    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885847    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885850    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885853    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885855    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885858    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885861    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885863    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885866    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885869    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885872    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:26.890187 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885875    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885877    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885881    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885885    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885888    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885891    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885893    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885896    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885899    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885902    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885905    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885907    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885910    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885913    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885916    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885918    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885921    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885923    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885926    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:26.890641 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885929    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885932    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885934    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885937    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885940    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885943    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885946    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885948    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885951    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885954    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885956    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885959    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885962    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885965    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885967    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885971    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885974    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.885977    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887137    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887147    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887154    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 14:16:26.891163 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887159    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887164    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887168    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887172    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887176    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887180    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887183    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887187    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887190    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887193    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887197    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887200    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887203    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887206    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887209    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887213    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887216    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887219    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887223    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887226    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887230    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887233    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887236    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 14:16:26.891708 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887242    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887246    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887250    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887253    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887257    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887260    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887265    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887268    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887271    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887274    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887277    2575 flags.go:64] FLAG: --enable-server="true"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887281    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887285    2575 flags.go:64] FLAG: --event-burst="100"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887289    2575 flags.go:64] FLAG: --event-qps="50"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887292    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887295    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887298    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887302    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887305    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887308    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887312    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887315    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887318    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887321    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887324    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 14:16:26.892245 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887327    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887330    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887333    2575 flags.go:64] FLAG: --feature-gates=""
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887338    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887341    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887344    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887348    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887356    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887360    2575 flags.go:64] FLAG: --help="false"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887363    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887367    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887370    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887373    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887376    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887380    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887395    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887399    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887402    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887405    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887408    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887412    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887415    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887418    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887421    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 14:16:26.892879 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887424    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887427    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887430    2575 flags.go:64] FLAG: --lock-file=""
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887433    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887436    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887439    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887445    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887448    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887451    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887454    2575 flags.go:64] FLAG: --logging-format="text"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887457    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887460    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887463    2575 flags.go:64] FLAG: --manifest-url=""
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887467    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887471    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887476    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887480    2575 flags.go:64] FLAG: --max-pods="110"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887483    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887486    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887490    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887493    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887497    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887500    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887503    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887510    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 14:16:26.893470 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887513    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887516    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887520    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887523    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887526    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887529    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887532    2575 flags.go:64] FLAG: --port="10250"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887535    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887538    2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-049690c63fd336593"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887541    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887544    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887547    2575 flags.go:64] FLAG: --register-node="true"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887550    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887554    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887557    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887560    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887563    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887567    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887570    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887573    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887577    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887580    2575 flags.go:64] FLAG: --runonce="false"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887584    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887588    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887591    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887594    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887597    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 14:16:26.894412 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887601    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887604    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887607    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887610    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887614    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887617    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887620    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887623    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887626    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887629    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887634    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887637    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887640    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887643    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887646    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887649    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887653    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887656    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887659    2575 flags.go:64] FLAG: --v="2"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887664    2575 flags.go:64] FLAG: --version="false"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887668    2575 flags.go:64] FLAG: --vmodule=""
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887672    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.887675    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887815    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:26.895570 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887819    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887824    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887827    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887831    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887834    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887836    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887839    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887842    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887844    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887847    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887850    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887853    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887856    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887859    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887862    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887865    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887867    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887870    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887873    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887876    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:26.896679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887878    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887881    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887884    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887886    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887889    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887892    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887894    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887898    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887902    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887906    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887909    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887912    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887915    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887919    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887922    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887927    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887929    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887932    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887935    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:26.897563 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887938    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887941    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887943    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887947    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887950    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887952    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887955    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887958    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887960    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887963    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887966    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887969    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887971    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887974    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887976    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887979    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887982    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887984    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887987    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887990    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:26.898381 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887992    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887995    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.887998    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888001    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888003    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888006    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888010    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888012    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888017    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888019    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888022    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888025    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888028    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888030    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888033    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888037    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888042    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888046    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888049    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:26.898889 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888052    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888054    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888057    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888059    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888062    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888065    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888067    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888070    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888072    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888075    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888078    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888080    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888083    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888085    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888088    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888091    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888093    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888096    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888099    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:26.899651 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888102    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888105    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888109    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888111    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888114    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888117    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888119    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888122    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.888150    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888190    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888195    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888198    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888203    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888208    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888211    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:26.900456 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888214    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888217    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888220    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888222    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888225    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888228    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888230    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888233    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888236    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888238    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888241    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888244    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888246    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888249    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888251    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888254    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888257    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888260    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888264    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:26.900842 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888268    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888271    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888274    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888276    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888279    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888282    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888285    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888288    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888290    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888293    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888296    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888299    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888302    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888304    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888307    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888310    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888312    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888315    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888317    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888320    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:26.901547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888322    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888325    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888328    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888331    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888333    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888336    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888339    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888342    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888345    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888348    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888351    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888354    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888358    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888361    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888364    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888366    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888369    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888374    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888377    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:26.902033 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888380    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888397    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888400    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888403    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888406    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888410    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888413    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888415    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888418    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888421    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888423    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888426    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888429    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888432    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888435    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888437    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888440    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888442    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888445    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:26.902547 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888447    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888450    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888453    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888455    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888458    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888462    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888465    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888468    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888470    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888473    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888475    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888478    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888481    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888483    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888486    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888489    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888492    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888494    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888497    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888500    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:26.902977 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888503    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888506    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.888509    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.896159    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.896176    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.896199    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.896206    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896290    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896298    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896306    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896311    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896315    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896320    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896325    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896329    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896334    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896339    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896343    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896347    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896352    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:26.903591 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896357    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896361    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896365    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896369    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896374    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896378    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896399    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896404    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896408    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896412    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896419    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896424    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896428    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896432    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896437    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896441    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896446    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896450    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896454    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896459    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:26.904059 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896464    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896468    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896473    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896477    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896482    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896486    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896490    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896495    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896499    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896504    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896508    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896512    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896517    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896521    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896525    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896530    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896534    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896539    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896543    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:26.904689 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896548    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896553    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896557    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896562    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896568    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896572    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896577    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896581    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896588    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896594    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896601    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896608    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896614    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896618    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896623    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896628    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896632    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896636    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:26.905120 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896641    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896645    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896650    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896654    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896659    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896663    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896667    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896672    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896676    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896680    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896684    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896688    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896692    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896696    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896700    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896704    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896709    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896714    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896719    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:26.905864 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896724    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896729    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896734    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896739    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896743    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896747    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896751    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896755    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896760    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896764    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896768    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896773    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896777    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896782    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896786    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896790    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:26.906654 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.896794    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.896837    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897086    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897095    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897101    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897106    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897110    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897115    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897120    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897124    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897129    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897133    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897138    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897142    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897148    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897152    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:26.907310 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897156    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897162    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897166    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897170    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897174    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897178    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897183    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897187    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897191    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897195    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897199    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897204    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897208    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897213    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897217    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897221    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897225    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897230    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897234    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:26.907792 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897239    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897243    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897247    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897252    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897256    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897260    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897267    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897273    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897277    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897282    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897288    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897293    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897297    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897302    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897307    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897311    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897315    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897319    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897324    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:26.908275 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897328    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897333    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897337    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897341    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897345    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897350    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897354    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897358    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897362    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897366    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897370    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897375    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897379    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897399    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897405    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897409    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897414    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897418    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897422    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897426    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:26.908732 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897431    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897435    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897441    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897446    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897451    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897455    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897460    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897464    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897469    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897474    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897478    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897483    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897487    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897491    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897495    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897499    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897504    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897508    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897513    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:26.909232 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897517    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897521    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897525    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897530    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897534    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897538    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897543    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897547    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897551    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897555    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897559    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897564    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897575    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897580    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897584    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.897627    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:26.909854 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897689    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897697    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897703    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897708    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897712    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897717    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897721    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897726    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897731    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897738    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897744    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897749    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897754    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897759    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897763    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897767    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897771    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897776    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897781    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:26.910231 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897785    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897790    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897794    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897799    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897803    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897807    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897811    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897815    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897820    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897824    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897828    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897833    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897837    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897841    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897845    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897850    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897854    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897859    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897864    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897869    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:26.910787 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897873    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897878    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897882    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897886    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897891    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897898    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897904    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897909    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897914    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897919    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897923    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897927    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897932    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897936    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897940    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897948    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897952    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897957    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897961    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:26.911264 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897966    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897970    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897974    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897979    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897983    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897987    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897992    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.897996    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898001    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898005    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898009    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898013    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898018    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898022    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898026    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898031    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898036    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898040    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898045    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:26.911772 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898049    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898053    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898058    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898062    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898066    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898071    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898075    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898079    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898083    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898092    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898096    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898102    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898106    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898110    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898115    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898119    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898124    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898128    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898133    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898137    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:26.912222 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898141    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898145    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898150    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898154    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898159    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898163    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898167    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898172    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:26.898176    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.898250    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.901079    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.902248    2575 server.go:1014] "Starting client certificate rotation"
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.902350    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.902397    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.905721    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 14:16:26.912767 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.907791    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 14:16:26.919095 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.919078    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 01 14:16:26.922375 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.922359    2575 log.go:25] "Validated CRI v1 image API"
Nov 01 14:16:26.923139 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.923114    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 14:16:26.926007 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.925989    2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9e00471d-7bdb-4dfa-8f33-835dc29ea08d:/dev/nvme0n1p4 fc362e37-508d-4f01-b43a-63b9653bbbb8:/dev/nvme0n1p3]
Nov 01 14:16:26.926076 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.926007    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 14:16:26.931200 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.931094    2575 manager.go:217] Machine: {Timestamp:2026-11-01 14:16:26.929619022 +0000 UTC m=+0.361152263 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100318 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc86f7c7b159b8da86698ee5d7280 SystemUUID:ec2fc86f-7c7b-159b-8da8-6698ee5d7280 BootID:57b27608-5a5a-41d1-944c-8b8f9696027c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c0:88:5f:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c0:88:5f:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:31:c2:4e:8c:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 14:16:26.931200 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.931199    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 14:16:26.931348 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.931336    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 14:16:26.932276 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.932252    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 14:16:26.932419 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.932276    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-142.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 14:16:26.932955 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.932946    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 14:16:26.932990 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.932956    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 14:16:26.932990 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.932976    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 14:16:26.932990 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.932988    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 14:16:26.933853 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.933843    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 14:16:26.937650 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.937637    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 14:16:26.937760 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.937751    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 14:16:26.938537 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.938528    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 14:16:26.938569 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.938541    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 14:16:26.938569 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.938552    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 14:16:26.938569 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.938560    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 01 14:16:26.938664 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.938579    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 14:16:26.938871 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.938857    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 14:16:26.941642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.941626    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 14:16:26.946006 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.945993    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 14:16:26.946789 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.946776    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 14:16:26.946837 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.946806    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 14:16:26.946951 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.946940    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 14:16:26.946988 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.946956    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 14:16:26.946988 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.946962    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 14:16:26.946988 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.946968    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 14:16:26.946988 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.946973    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 14:16:26.946988 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.946979    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 14:16:26.947600 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.947592    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 14:16:26.947638 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.947601    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 14:16:26.947638 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.947609    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 14:16:26.947638 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.947615    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 14:16:26.947638 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.947623    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 14:16:26.947638 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.947632    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 14:16:26.948506 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.948496    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 14:16:26.948506 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.948506    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 14:16:26.952124 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.952113    2575 server.go:1263] "Started kubelet"
Nov 01 14:16:26.952198 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.952173    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 14:16:26.952246 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.952206    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 14:16:26.952301 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.952290    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 14:16:26.952786 ip-10-0-135-142 systemd[1]: Started Kubernetes Kubelet.
Nov 01 14:16:26.955285 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.955248    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 14:16:26.955364 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.955337    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 14:16:26.959128 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.959111    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 14:16:26.959332 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:26.959317    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 14:16:26.959550 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.959535    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 14:16:26.959731 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:26.959703    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:26.960295 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960274    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 14:16:26.960404 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960305    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 14:16:26.960404 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960320    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 14:16:26.960661 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960648    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 14:16:26.960661 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960661    2575 factory.go:55] Registering systemd factory
Nov 01 14:16:26.960736 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960667    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 01 14:16:26.960736 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960666    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 14:16:26.960736 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960686    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 14:16:26.960912 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960898    2575 factory.go:153] Registering CRI-O factory
Nov 01 14:16:26.960970 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960915    2575 factory.go:223] Registration of the crio container factory successfully
Nov 01 14:16:26.960970 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960947    2575 factory.go:103] Registering Raw factory
Nov 01 14:16:26.960970 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.960961    2575 manager.go:1199] Started watching for new ooms in manager
Nov 01 14:16:26.962195 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.962166    2575 manager.go:319] Starting recovery of all containers
Nov 01 14:16:26.968025 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.967996    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 14:16:26.975572 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.975547    2575 manager.go:324] Recovery completed
Nov 01 14:16:26.980781 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.980766    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:26.982483 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.982465    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:26.982564 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.982494    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:26.982564 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.982504    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:26.983054 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.983040    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 01 14:16:26.983054 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.983053    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 14:16:26.983156 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.983044    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:26.983156 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.983073    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 14:16:26.983784 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.983768    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:26.983843 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.983801    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:26.983843 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.983816    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:26.985306 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.985290    2575 policy_none.go:50] "Start"
Nov 01 14:16:26.985759 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.985744    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 14:16:26.985793 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.985772    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 14:16:26.987917 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:26.987904    2575 policy_none.go:44] "Start"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.029841    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.032257    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.032264    2575 manager.go:341] "Starting Device Plugin manager"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.032272    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.032701    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.032745    2575 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.032892    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.032907    2575 server.go:87] "Starting device plugin registration server"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.033158    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.033169    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.033299    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.033367    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.033374    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.033805    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 14:16:27.047356 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.033839    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:27.092313 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.092293    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-142.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 14:16:27.092448 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.092294    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-142.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 14:16:27.093398 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.092353    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-142.us-west-2.compute.internal.18e3f17af0d07c80  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-142.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-142.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:26.952088704 +0000 UTC m=+0.383621945,LastTimestamp:2026-11-01 14:16:26.952088704 +0000 UTC m=+0.383621945,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-142.us-west-2.compute.internal,}"
Nov 01 14:16:27.133727 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.133690    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:27.134213 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.134195    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"]
Nov 01 14:16:27.134301 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.134290    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:27.134850 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.134817    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:27.134937 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.134855    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:27.134937 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.134869    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:27.134937 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.134899    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.135095 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.135076    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:27.135178 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.135108    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:27.135178 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.135130    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:27.137233 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137220    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:27.137372 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137357    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.137452 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137440    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:27.137873 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137855    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:27.137873 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137863    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:27.137996 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137882    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:27.137996 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137893    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:27.137996 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137884    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:27.137996 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.137962    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:27.139908 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.139892    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.139995 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.139914    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:27.140609 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.140590    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:27.140698 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.140625    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:27.140698 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.140639    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:27.160625 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.160606    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 14:16:27.160701 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.160662    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4rzbs"
Nov 01 14:16:27.161657 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.161641    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.161718 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.161666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.161718 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.161683    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccbcd60b09b5d4721c0b990d6d1ecb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"fccbcd60b09b5d4721c0b990d6d1ecb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.180753 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.180733    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found" node="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.182442 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.182358    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-142.us-west-2.compute.internal.18e3f17af2a03f86  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-142.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-142.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-142.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:26.982481798 +0000 UTC m=+0.414015038,LastTimestamp:2026-11-01 14:16:26.982481798 +0000 UTC m=+0.414015038,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-142.us-west-2.compute.internal,}"
Nov 01 14:16:27.182704 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.182683    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 14:16:27.182744 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.182684    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 01 14:16:27.183689 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.183669    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found" node="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.229463 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.229434    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 14:16:27.229537 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.229477    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-142.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 14:16:27.243706 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.243689    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.243756 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.243708    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-142.us-west-2.compute.internal\": node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:27.262796 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.262776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.262858 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.262802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.262858 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.262824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccbcd60b09b5d4721c0b990d6d1ecb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"fccbcd60b09b5d4721c0b990d6d1ecb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.262922 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.262875    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccbcd60b09b5d4721c0b990d6d1ecb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"fccbcd60b09b5d4721c0b990d6d1ecb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.262922 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.262880    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.262922 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.262875    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.359876 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.359853    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:27.460685 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.460639    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:27.483734 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.483720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.486885 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.486861    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:27.561832 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.561714    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:27.662166 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.662131    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:27.762581 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.762553    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:27.863366 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.863319    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:27.901969 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:27.901938    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 14:16:27.902483 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.902102    2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2ba49ef185a5444dbb27d29e6baf327-c3ad9244eeee9bec.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-142.us-west-2.compute.internal.18e3f17af2a0a523\": read tcp 10.0.135.142:37634->18.220.90.157:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-142.us-west-2.compute.internal.18e3f17af2a0a523  default   5179 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-142.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-142.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-142.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:26 +0000 UTC,LastTimestamp:2026-11-01 14:16:27.135134692 +0000 UTC m=+0.566667933,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-142.us-west-2.compute.internal,}"
Nov 01 14:16:27.963593 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:27.963558    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:28.064358 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:28.064281    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:28.165244 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:28.165199    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:28.165244 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.165210    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 14:11:27 +0000 UTC" deadline="2028-07-19 12:23:52.60180059 +0000 UTC"
Nov 01 14:16:28.165244 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.165243    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15022h7m24.436562655s"
Nov 01 14:16:28.259298 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.259257    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:28.260358 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.260342    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:28.260468 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.260374    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:28.260468 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.260386    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:28.265899 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:28.265882    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:28.288042 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.288022    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:28.366894 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:28.366818    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:28.467742 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:28.467709    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:16:28.489131 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.489107    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:28.559779 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.559738    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:28.620380 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.620319    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:16:28.622049 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.622033    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:16:28.640525 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.640498    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:28.680002 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.679961    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:16:28.777343 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.777314    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:28.801205 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:28.801166    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccbcd60b09b5d4721c0b990d6d1ecb1.slice/crio-4f95137e087a14eb0b1b186e49e0de878f9e03c618bfd3eafb035496c1b50fb0 WatchSource:0}: Error finding container 4f95137e087a14eb0b1b186e49e0de878f9e03c618bfd3eafb035496c1b50fb0: Status 404 returned error can't find the container with id 4f95137e087a14eb0b1b186e49e0de878f9e03c618bfd3eafb035496c1b50fb0
Nov 01 14:16:28.801492 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:28.801472    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e27d7e3a4b8b3b59f6de6ab62a43ca.slice/crio-c7d6abce1d36443e2f8e24fb0f4f96fe2ca3867067b40a422373fecd8ad0ac1a WatchSource:0}: Error finding container c7d6abce1d36443e2f8e24fb0f4f96fe2ca3867067b40a422373fecd8ad0ac1a: Status 404 returned error can't find the container with id c7d6abce1d36443e2f8e24fb0f4f96fe2ca3867067b40a422373fecd8ad0ac1a
Nov 01 14:16:28.807208 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.807192    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:16:28.941593 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.941512    2575 apiserver.go:52] "Watching apiserver"
Nov 01 14:16:28.961137 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:28.961112    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 14:16:29.039137 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.039085    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"fccbcd60b09b5d4721c0b990d6d1ecb1","Type":"ContainerStarted","Data":"4f95137e087a14eb0b1b186e49e0de878f9e03c618bfd3eafb035496c1b50fb0"}
Nov 01 14:16:29.040246 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.040222    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"38e27d7e3a4b8b3b59f6de6ab62a43ca","Type":"ContainerStarted","Data":"c7d6abce1d36443e2f8e24fb0f4f96fe2ca3867067b40a422373fecd8ad0ac1a"}
Nov 01 14:16:29.044794 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.044777    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 14:16:29.045103 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.045087    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-g44v5","openshift-cluster-node-tuning-operator/tuned-4slbg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f","openshift-network-diagnostics/network-check-target-thrgt","openshift-network-operator/iptables-alerter-qfz2f","kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal","openshift-image-registry/node-ca-2nh9b","openshift-multus/multus-additional-cni-plugins-86fr6","openshift-multus/multus-wn9x6","openshift-dns/node-resolver-tqbqm","openshift-multus/network-metrics-daemon-9c6pv","openshift-ovn-kubernetes/ovnkube-node-k8d4s","kube-system/global-pull-secret-syncer-z9bf5"]
Nov 01 14:16:29.049006 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.048988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.051328 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.051307    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.053720 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.053703    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.053814 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.053763    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:29.057496 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.057478    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.060235 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.060124    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:29.060330 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.060246    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:29.063337 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.063281    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.065793 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.065771    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.068163 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.068143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.070570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.070551    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.072800 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.072780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:29.073304 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073286    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-conf-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.073403 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-daemon-config\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.073403 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-kubelet\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073403 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073351    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-systemd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073585 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-ovn\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073585 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073457    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073585 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.073585 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-899qc\" (UniqueName: \"kubernetes.io/projected/881f3467-ef5c-4bcc-b730-5f07644265b6-kube-api-access-899qc\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.073585 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-multus\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.073585 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-netns\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073606    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-bin\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-env-overrides\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073682    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-device-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-netd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08c55c12-d679-4874-9e6f-789e7829641c-ovn-node-metrics-cert\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-script-lib\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073833    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfk4t\" (UniqueName: \"kubernetes.io/projected/08c55c12-d679-4874-9e6f-789e7829641c-kube-api-access-sfk4t\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.073888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-socket-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-multus-certs\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-dbus\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.073995    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29kd\" (UniqueName: \"kubernetes.io/projected/4681a549-d1f9-4bed-82c9-21a214c885b6-kube-api-access-n29kd\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/881f3467-ef5c-4bcc-b730-5f07644265b6-iptables-alerter-script\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-socket-dir-parent\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074077    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-bin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-var-lib-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-sys-fs\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/881f3467-ef5c-4bcc-b730-5f07644265b6-host-slash\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-system-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cni-binary-copy\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074231    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-kubelet\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-systemd-units\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-config\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.074451 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074330    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-registration-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9zk\" (UniqueName: \"kubernetes.io/projected/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-kube-api-access-qj9zk\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-slash\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-etc-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-node-log\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074469    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-log-socket\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074506    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-hostroot\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-kubelet-config\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cnibin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-k8s-cni-cncf-io\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-netns\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-etc-kubernetes\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.074678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.075256 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.075018    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.077352 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.077331    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:29.077475 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.077397    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:29.101999 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.101963    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 14:16:29.102107 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.102048    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 14:16:29.102107 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.102049    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 14:16:29.102107 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.102070    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj7jb\""
Nov 01 14:16:29.102107 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.102082    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 14:16:29.103323 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.103299    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 14:16:29.103423 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.103407    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 14:16:29.103516 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.103421    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 14:16:29.103516 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.103487    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fnkb\""
Nov 01 14:16:29.103632 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.103560    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 14:16:29.103760 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.103740    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 14:16:29.103894 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.103882    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 14:16:29.109158 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.109136    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 14:16:29.109238 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.109207    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 14:16:29.109357 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.109332    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lp9km\""
Nov 01 14:16:29.109479 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.109357    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 14:16:29.115538 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.115521    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 14:16:29.115608 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.115558    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:29.116224 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.116210    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m46fn\""
Nov 01 14:16:29.116294 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.116223    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:29.117255 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.117241    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 14:16:29.117309 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.117288    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 14:16:29.117575 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.117561    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-24rb9\""
Nov 01 14:16:29.120134 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.120118    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 14:16:29.120205 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.120136    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 14:16:29.120205 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.120171    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 14:16:29.120266 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.120210    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v4966\""
Nov 01 14:16:29.147925 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.147904    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 14:16:29.148016 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.147968    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4lb6c"
Nov 01 14:16:29.161792 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.161775    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 14:16:29.175076 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-systemd\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.175161 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175081    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wrs\" (UniqueName: \"kubernetes.io/projected/14c45109-a849-4e82-9c13-c5d1af56806e-kube-api-access-g4wrs\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.175161 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jlct\" (UniqueName: \"kubernetes.io/projected/f9f644ed-0d59-4762-af3b-45eefe8edc78-kube-api-access-7jlct\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:29.175161 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175132    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-registration-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.175299 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-slash\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175299 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175216    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-registration-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.175299 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-log-socket\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175299 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175263    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-slash\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175299 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175262    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-log-socket\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175299 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysconfig\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.175601 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175601 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-script-lib\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175601 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-kubelet-config\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.175601 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175434    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175601 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175492    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-kubelet-config\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.175601 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175551    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:29.175601 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175581    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cnibin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-etc-kubernetes\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-ovn\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-tmp\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175686    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-etc-kubernetes\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-conf-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175713    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cnibin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175722    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-ovn\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-kubelet\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175778    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-conf-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/120f88bc-d7ec-43a0-8503-726fb3ca3947-agent-certs\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175829    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-kubelet\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.175889 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175917    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-script-lib\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.175935    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899qc\" (UniqueName: \"kubernetes.io/projected/881f3467-ef5c-4bcc-b730-5f07644265b6-kube-api-access-899qc\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-bin\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176016    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-system-cni-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176076    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-bin\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176110    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-device-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176170    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-device-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-netd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176210    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-netd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08c55c12-d679-4874-9e6f-789e7829641c-ovn-node-metrics-cert\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-lib-modules\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-socket-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-cnibin\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176328    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-run\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.176453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-tuned\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176432    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-socket-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/881f3467-ef5c-4bcc-b730-5f07644265b6-iptables-alerter-script\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-socket-dir-parent\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-bin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176563    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-var-lib-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176623    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-bin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176673    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-socket-dir-parent\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-modprobe-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-var-lib-kubelet\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176732    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-var-lib-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-sys-fs\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/881f3467-ef5c-4bcc-b730-5f07644265b6-host-slash\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176799    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-sys-fs\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.177125 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176819    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/881f3467-ef5c-4bcc-b730-5f07644265b6-host-slash\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-system-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176873    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cni-binary-copy\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-kubelet\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176898    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/881f3467-ef5c-4bcc-b730-5f07644265b6-iptables-alerter-script\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176925    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-systemd-units\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176937    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-system-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176966    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-kubelet\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176972    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-systemd-units\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177077    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.176948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-config\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9zk\" (UniqueName: \"kubernetes.io/projected/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-kube-api-access-qj9zk\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-etc-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-node-log\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-binary-copy\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177631    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-etc-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.177783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177636    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-node-log\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177636    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfs4\" (UniqueName: \"kubernetes.io/projected/57920aa2-346d-4bc2-99a0-6e92f382fa30-kube-api-access-8mfs4\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177657    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cni-binary-copy\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-hostroot\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177754    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-k8s-cni-cncf-io\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177793    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-k8s-cni-cncf-io\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177793    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-netns\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177805    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-hostroot\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177861    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-netns\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14c45109-a849-4e82-9c13-c5d1af56806e-serviceca\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177916    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-sys\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177946    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57920aa2-346d-4bc2-99a0-6e92f382fa30-hosts-file\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.177971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57920aa2-346d-4bc2-99a0-6e92f382fa30-tmp-dir\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-daemon-config\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178098    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-config\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178152    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-systemd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-systemd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.178673 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178282    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178364    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5ws4\" (UniqueName: \"kubernetes.io/projected/746ff76c-dd84-49d4-b178-c687185af620-kube-api-access-b5ws4\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/120f88bc-d7ec-43a0-8503-726fb3ca3947-konnectivity-ca\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-multus\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-netns\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-env-overrides\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178550    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-netns\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178577    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-multus\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178591    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-daemon-config\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-kubernetes\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178640    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-conf\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.178980    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfk4t\" (UniqueName: \"kubernetes.io/projected/08c55c12-d679-4874-9e6f-789e7829641c-kube-api-access-sfk4t\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-host\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.179431 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179276    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-env-overrides\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-multus-certs\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179482    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rlx6\" (UniqueName: \"kubernetes.io/projected/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-kube-api-access-8rlx6\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-dbus\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.179591    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29kd\" (UniqueName: \"kubernetes.io/projected/4681a549-d1f9-4bed-82c9-21a214c885b6-kube-api-access-n29kd\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c45109-a849-4e82-9c13-c5d1af56806e-host\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.179666    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret podName:87bcc491-1e33-442b-b230-0cb7ae30f544 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:29.679644762 +0000 UTC m=+3.111178008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret") pod "global-pull-secret-syncer-z9bf5" (UID: "87bcc491-1e33-442b-b230-0cb7ae30f544") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179746    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179814    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-dbus\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.180280 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.179815    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-multus-certs\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.182160 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.182140    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08c55c12-d679-4874-9e6f-789e7829641c-ovn-node-metrics-cert\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.210778 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.210729    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 14:16:29.210869 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.210807    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 14:16:29.210869 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.210837    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:29.210869 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.210838    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7pw86\""
Nov 01 14:16:29.210982 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.210932    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 14:16:29.210982 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.210968    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-smpqh\""
Nov 01 14:16:29.210982 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.210975    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:29.211090 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.211007    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 14:16:29.211090 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.211007    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9p5x8\""
Nov 01 14:16:29.230553 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.230533    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:29.230553 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.230553    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:29.230655 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.230564    2575 projected.go:196] Error preparing data for projected volume kube-api-access-xxhn8 for pod openshift-network-diagnostics/network-check-target-thrgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:29.230655 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.230621    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8 podName:ac04470f-39e4-48f5-8d71-364e1d95f644 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:29.730606367 +0000 UTC m=+3.162139613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xxhn8" (UniqueName: "kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8") pod "network-check-target-thrgt" (UID: "ac04470f-39e4-48f5-8d71-364e1d95f644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:29.232949 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.232920    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-899qc\" (UniqueName: \"kubernetes.io/projected/881f3467-ef5c-4bcc-b730-5f07644265b6-kube-api-access-899qc\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.233766 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.233742    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9zk\" (UniqueName: \"kubernetes.io/projected/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-kube-api-access-qj9zk\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.235643 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.235624    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfk4t\" (UniqueName: \"kubernetes.io/projected/08c55c12-d679-4874-9e6f-789e7829641c-kube-api-access-sfk4t\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.235797 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.235779    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29kd\" (UniqueName: \"kubernetes.io/projected/4681a549-d1f9-4bed-82c9-21a214c885b6-kube-api-access-n29kd\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.279972 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.279950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.280070 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.279978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-kubernetes\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280070 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.279995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-conf\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280070 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-host\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280070 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rlx6\" (UniqueName: \"kubernetes.io/projected/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-kube-api-access-8rlx6\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280070 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c45109-a849-4e82-9c13-c5d1af56806e-host\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280074    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-kubernetes\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-systemd\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280102    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-host\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wrs\" (UniqueName: \"kubernetes.io/projected/14c45109-a849-4e82-9c13-c5d1af56806e-kube-api-access-g4wrs\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280142    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jlct\" (UniqueName: \"kubernetes.io/projected/f9f644ed-0d59-4762-af3b-45eefe8edc78-kube-api-access-7jlct\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280152    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c45109-a849-4e82-9c13-c5d1af56806e-host\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysconfig\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-tmp\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280230    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-systemd\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280267 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/120f88bc-d7ec-43a0-8503-726fb3ca3947-agent-certs\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280168    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-conf\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-system-cni-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280344    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysconfig\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-lib-modules\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280370    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280394    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-system-cni-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-cnibin\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-run\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-tuned\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280500    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-cnibin\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280528    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280536    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-lib-modules\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280563    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-run\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-modprobe-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.280642 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.280630    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280658    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-var-lib-kubelet\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-var-lib-kubelet\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.280693    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs podName:f9f644ed-0d59-4762-af3b-45eefe8edc78 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:29.780676447 +0000 UTC m=+3.212209692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs") pod "network-metrics-daemon-9c6pv" (UID: "f9f644ed-0d59-4762-af3b-45eefe8edc78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-binary-copy\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfs4\" (UniqueName: \"kubernetes.io/projected/57920aa2-346d-4bc2-99a0-6e92f382fa30-kube-api-access-8mfs4\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280733    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-modprobe-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14c45109-a849-4e82-9c13-c5d1af56806e-serviceca\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-sys\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.280965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57920aa2-346d-4bc2-99a0-6e92f382fa30-hosts-file\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57920aa2-346d-4bc2-99a0-6e92f382fa30-tmp-dir\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281038    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ws4\" (UniqueName: \"kubernetes.io/projected/746ff76c-dd84-49d4-b178-c687185af620-kube-api-access-b5ws4\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/120f88bc-d7ec-43a0-8503-726fb3ca3947-konnectivity-ca\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281121    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57920aa2-346d-4bc2-99a0-6e92f382fa30-hosts-file\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281139    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.281390 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281182    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-sys\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.281948 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281409    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.281948 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281480    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57920aa2-346d-4bc2-99a0-6e92f382fa30-tmp-dir\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.281948 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281546    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.281948 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281556    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14c45109-a849-4e82-9c13-c5d1af56806e-serviceca\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.281948 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281740    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/120f88bc-d7ec-43a0-8503-726fb3ca3947-konnectivity-ca\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:29.281948 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.281922    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-binary-copy\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.283366 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.283334    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/120f88bc-d7ec-43a0-8503-726fb3ca3947-agent-certs\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:29.283439 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.283422    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-tuned\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.283501 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.283490    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-tmp\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.337738 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.337715    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wrs\" (UniqueName: \"kubernetes.io/projected/14c45109-a849-4e82-9c13-c5d1af56806e-kube-api-access-g4wrs\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.338045 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.338004    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfs4\" (UniqueName: \"kubernetes.io/projected/57920aa2-346d-4bc2-99a0-6e92f382fa30-kube-api-access-8mfs4\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.338136 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.338090    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jlct\" (UniqueName: \"kubernetes.io/projected/f9f644ed-0d59-4762-af3b-45eefe8edc78-kube-api-access-7jlct\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:29.338397 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.338373    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ws4\" (UniqueName: \"kubernetes.io/projected/746ff76c-dd84-49d4-b178-c687185af620-kube-api-access-b5ws4\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.338432 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.338376    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rlx6\" (UniqueName: \"kubernetes.io/projected/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-kube-api-access-8rlx6\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.361407 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.361383    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wn9x6"
Nov 01 14:16:29.371610 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.371586    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f945e0d_6eb2_4c2f_ac70_e37fffe05749.slice/crio-4b0517da2462b1dce1e4b7a0f268fa4dddf46af677f7926f9062e8a330543399 WatchSource:0}: Error finding container 4b0517da2462b1dce1e4b7a0f268fa4dddf46af677f7926f9062e8a330543399: Status 404 returned error can't find the container with id 4b0517da2462b1dce1e4b7a0f268fa4dddf46af677f7926f9062e8a330543399
Nov 01 14:16:29.387830 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.387809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:16:29.390841 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.390734    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:16:29.398191 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.398172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:29.398496 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.398472    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881f3467_ef5c_4bcc_b730_5f07644265b6.slice/crio-0bdc2b910c56d063d994085516c825f7efa30056144d57f46a0a7a969ceccbb2 WatchSource:0}: Error finding container 0bdc2b910c56d063d994085516c825f7efa30056144d57f46a0a7a969ceccbb2: Status 404 returned error can't find the container with id 0bdc2b910c56d063d994085516c825f7efa30056144d57f46a0a7a969ceccbb2
Nov 01 14:16:29.400297 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.400278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:16:29.400420 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.400388    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4681a549_d1f9_4bed_82c9_21a214c885b6.slice/crio-37d0c0308a1a50f4e13dff6538ed02d2c3a5b704cdbfb225f8a678a78870f2b2 WatchSource:0}: Error finding container 37d0c0308a1a50f4e13dff6538ed02d2c3a5b704cdbfb225f8a678a78870f2b2: Status 404 returned error can't find the container with id 37d0c0308a1a50f4e13dff6538ed02d2c3a5b704cdbfb225f8a678a78870f2b2
Nov 01 14:16:29.407714 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.407694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:16:29.408635 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.408616    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c55c12_d679_4874_9e6f_789e7829641c.slice/crio-9bf76f71fb7565eb8d5c13c0ee8f717273f4bae30fbcbd85fa429fcfc9882cce WatchSource:0}: Error finding container 9bf76f71fb7565eb8d5c13c0ee8f717273f4bae30fbcbd85fa429fcfc9882cce: Status 404 returned error can't find the container with id 9bf76f71fb7565eb8d5c13c0ee8f717273f4bae30fbcbd85fa429fcfc9882cce
Nov 01 14:16:29.409837 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.409821    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:29.410355 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.410330    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c45109_a849_4e82_9c13_c5d1af56806e.slice/crio-6c201adaea417dcb703e8ca6f2cc5c2ceb78c2fb48a9b29f40921ac87df2020d WatchSource:0}: Error finding container 6c201adaea417dcb703e8ca6f2cc5c2ceb78c2fb48a9b29f40921ac87df2020d: Status 404 returned error can't find the container with id 6c201adaea417dcb703e8ca6f2cc5c2ceb78c2fb48a9b29f40921ac87df2020d
Nov 01 14:16:29.418184 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.418078    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:16:29.418351 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.418326    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a11f89e_8c0d_4412_999f_b73cd5d20e2d.slice/crio-4b3786a9b968e8e5b90c3fb208c0ee4939f89f2ed1885027630d1a5aab53ae51 WatchSource:0}: Error finding container 4b3786a9b968e8e5b90c3fb208c0ee4939f89f2ed1885027630d1a5aab53ae51: Status 404 returned error can't find the container with id 4b3786a9b968e8e5b90c3fb208c0ee4939f89f2ed1885027630d1a5aab53ae51
Nov 01 14:16:29.420094 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.420077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:16:29.420332 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.420312    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120f88bc_d7ec_43a0_8503_726fb3ca3947.slice/crio-1cfc1eb69dff98fe4a10b20fd76e3c5b961792435b474c2c02de16236cc7aeee WatchSource:0}: Error finding container 1cfc1eb69dff98fe4a10b20fd76e3c5b961792435b474c2c02de16236cc7aeee: Status 404 returned error can't find the container with id 1cfc1eb69dff98fe4a10b20fd76e3c5b961792435b474c2c02de16236cc7aeee
Nov 01 14:16:29.427679 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.427656    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746ff76c_dd84_49d4_b178_c687185af620.slice/crio-728bb1a9335a76ec4d258e7fae981ec3740a755af555a1b2fd2ce57b8c84d3d9 WatchSource:0}: Error finding container 728bb1a9335a76ec4d258e7fae981ec3740a755af555a1b2fd2ce57b8c84d3d9: Status 404 returned error can't find the container with id 728bb1a9335a76ec4d258e7fae981ec3740a755af555a1b2fd2ce57b8c84d3d9
Nov 01 14:16:29.430435 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:16:29.430413    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57920aa2_346d_4bc2_99a0_6e92f382fa30.slice/crio-6e9418e6ba607b400c5f12d970b7e59e9b54fc48a314098b0a4afcd1c9e95d7e WatchSource:0}: Error finding container 6e9418e6ba607b400c5f12d970b7e59e9b54fc48a314098b0a4afcd1c9e95d7e: Status 404 returned error can't find the container with id 6e9418e6ba607b400c5f12d970b7e59e9b54fc48a314098b0a4afcd1c9e95d7e
Nov 01 14:16:29.686744 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.686652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:29.686887 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.686761    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:29.686887 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.686816    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret podName:87bcc491-1e33-442b-b230-0cb7ae30f544 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:30.686798173 +0000 UTC m=+4.118331416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret") pod "global-pull-secret-syncer-z9bf5" (UID: "87bcc491-1e33-442b-b230-0cb7ae30f544") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:29.787401 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.787366    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:29.787610 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:29.787438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:29.787610 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.787542    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:29.787610 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.787564    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:29.787610 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.787574    2575 projected.go:196] Error preparing data for projected volume kube-api-access-xxhn8 for pod openshift-network-diagnostics/network-check-target-thrgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:29.787610 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.787596    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:29.787840 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.787628    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8 podName:ac04470f-39e4-48f5-8d71-364e1d95f644 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:30.787611432 +0000 UTC m=+4.219144665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xxhn8" (UniqueName: "kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8") pod "network-check-target-thrgt" (UID: "ac04470f-39e4-48f5-8d71-364e1d95f644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:29.787840 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:29.787646    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs podName:f9f644ed-0d59-4762-af3b-45eefe8edc78 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:30.787639207 +0000 UTC m=+4.219172441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs") pod "network-metrics-daemon-9c6pv" (UID: "f9f644ed-0d59-4762-af3b-45eefe8edc78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:30.044495 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.043983    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tqbqm" event={"ID":"57920aa2-346d-4bc2-99a0-6e92f382fa30","Type":"ContainerStarted","Data":"6e9418e6ba607b400c5f12d970b7e59e9b54fc48a314098b0a4afcd1c9e95d7e"}
Nov 01 14:16:30.045898 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.045865    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g44v5" event={"ID":"120f88bc-d7ec-43a0-8503-726fb3ca3947","Type":"ContainerStarted","Data":"1cfc1eb69dff98fe4a10b20fd76e3c5b961792435b474c2c02de16236cc7aeee"}
Nov 01 14:16:30.047678 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.047613    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f" event={"ID":"4681a549-d1f9-4bed-82c9-21a214c885b6","Type":"ContainerStarted","Data":"37d0c0308a1a50f4e13dff6538ed02d2c3a5b704cdbfb225f8a678a78870f2b2"}
Nov 01 14:16:30.050696 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.050662    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerStarted","Data":"728bb1a9335a76ec4d258e7fae981ec3740a755af555a1b2fd2ce57b8c84d3d9"}
Nov 01 14:16:30.052736 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.052704    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4slbg" event={"ID":"1a11f89e-8c0d-4412-999f-b73cd5d20e2d","Type":"ContainerStarted","Data":"4b3786a9b968e8e5b90c3fb208c0ee4939f89f2ed1885027630d1a5aab53ae51"}
Nov 01 14:16:30.054260 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.054235    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2nh9b" event={"ID":"14c45109-a849-4e82-9c13-c5d1af56806e","Type":"ContainerStarted","Data":"6c201adaea417dcb703e8ca6f2cc5c2ceb78c2fb48a9b29f40921ac87df2020d"}
Nov 01 14:16:30.056107 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.056085    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"9bf76f71fb7565eb8d5c13c0ee8f717273f4bae30fbcbd85fa429fcfc9882cce"}
Nov 01 14:16:30.057661 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.057628    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qfz2f" event={"ID":"881f3467-ef5c-4bcc-b730-5f07644265b6","Type":"ContainerStarted","Data":"0bdc2b910c56d063d994085516c825f7efa30056144d57f46a0a7a969ceccbb2"}
Nov 01 14:16:30.058968 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.058941    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wn9x6" event={"ID":"2f945e0d-6eb2-4c2f-ac70-e37fffe05749","Type":"ContainerStarted","Data":"4b0517da2462b1dce1e4b7a0f268fa4dddf46af677f7926f9062e8a330543399"}
Nov 01 14:16:30.151711 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.151647    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:29 +0000 UTC" deadline="2028-08-12 04:17:24.491126334 +0000 UTC"
Nov 01 14:16:30.151711 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.151707    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15590h0m54.339423246s"
Nov 01 14:16:30.694588 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.694540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:30.694841 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:30.694816    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:30.694939 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:30.694891    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret podName:87bcc491-1e33-442b-b230-0cb7ae30f544 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:32.694872014 +0000 UTC m=+6.126405261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret") pod "global-pull-secret-syncer-z9bf5" (UID: "87bcc491-1e33-442b-b230-0cb7ae30f544") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:30.795897 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.795848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:30.796070 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:30.795921    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:30.796140 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:30.796075    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:30.796176 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:30.796140    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs podName:f9f644ed-0d59-4762-af3b-45eefe8edc78 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:32.796121281 +0000 UTC m=+6.227654525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs") pod "network-metrics-daemon-9c6pv" (UID: "f9f644ed-0d59-4762-af3b-45eefe8edc78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:30.796264 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:30.796249    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:30.796315 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:30.796272    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:30.796315 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:30.796286    2575 projected.go:196] Error preparing data for projected volume kube-api-access-xxhn8 for pod openshift-network-diagnostics/network-check-target-thrgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:30.796418 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:30.796330    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8 podName:ac04470f-39e4-48f5-8d71-364e1d95f644 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:32.79631574 +0000 UTC m=+6.227848985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xxhn8" (UniqueName: "kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8") pod "network-check-target-thrgt" (UID: "ac04470f-39e4-48f5-8d71-364e1d95f644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:31.036338 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:31.036264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:31.036514 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:31.036405    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:31.036587 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:31.036513    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:31.036654 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:31.036588    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:31.036654 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:31.036651    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:31.036767 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:31.036738    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:31.151885 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:31.151835    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:29 +0000 UTC" deadline="2028-05-02 02:36:30.594410959 +0000 UTC"
Nov 01 14:16:31.151885 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:31.151882    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13140h19m59.442534142s"
Nov 01 14:16:32.713011 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:32.712977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:32.713562 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:32.713132    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:32.713562 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:32.713203    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret podName:87bcc491-1e33-442b-b230-0cb7ae30f544 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:36.713183086 +0000 UTC m=+10.144716322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret") pod "global-pull-secret-syncer-z9bf5" (UID: "87bcc491-1e33-442b-b230-0cb7ae30f544") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:32.813940 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:32.813888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:32.814100 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:32.813983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:32.814183 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:32.814136    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:32.814183 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:32.814158    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:32.814183 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:32.814173    2575 projected.go:196] Error preparing data for projected volume kube-api-access-xxhn8 for pod openshift-network-diagnostics/network-check-target-thrgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:32.814344 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:32.814236    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8 podName:ac04470f-39e4-48f5-8d71-364e1d95f644 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:36.814216586 +0000 UTC m=+10.245749874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xxhn8" (UniqueName: "kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8") pod "network-check-target-thrgt" (UID: "ac04470f-39e4-48f5-8d71-364e1d95f644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:32.814344 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:32.814315    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:32.814476 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:32.814351    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs podName:f9f644ed-0d59-4762-af3b-45eefe8edc78 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:36.814339116 +0000 UTC m=+10.245872395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs") pod "network-metrics-daemon-9c6pv" (UID: "f9f644ed-0d59-4762-af3b-45eefe8edc78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:33.036143 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:33.036066    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:33.036346 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:33.036319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:33.036346 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:33.036223    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:33.036550 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:33.036425    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:33.036550 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:33.036319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:33.036669 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:33.036546    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:34.099254 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:34.098910    2575 generic.go:356] "Generic (PLEG): container finished" podID="38e27d7e3a4b8b3b59f6de6ab62a43ca" containerID="ff65d7b312a5af58db60f292c4b0e9ce8b279ee0996090f75e181102d162ffa0" exitCode=0
Nov 01 14:16:34.099254 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:34.099125    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"38e27d7e3a4b8b3b59f6de6ab62a43ca","Type":"ContainerDied","Data":"ff65d7b312a5af58db60f292c4b0e9ce8b279ee0996090f75e181102d162ffa0"}
Nov 01 14:16:34.110807 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:34.110775    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"fccbcd60b09b5d4721c0b990d6d1ecb1","Type":"ContainerStarted","Data":"518a174d356f638577498f293de3cb9f37c33bff487dd7bb16a1af03d14cdafa"}
Nov 01 14:16:34.116281 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:34.116252    2575 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="9e6faa3b2251b1c7b6daeb6f8eb03fb1ed69d785fa66b219fffac250f16cb9ea" exitCode=0
Nov 01 14:16:34.116409 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:34.116295    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerDied","Data":"9e6faa3b2251b1c7b6daeb6f8eb03fb1ed69d785fa66b219fffac250f16cb9ea"}
Nov 01 14:16:35.035476 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:35.034503    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:35.035476 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:35.034610    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:35.035476 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:35.034671    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:35.035476 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:35.034717    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:35.035476 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:35.034756    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:35.035476 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:35.034819    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:35.122026 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:35.121990    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g44v5" event={"ID":"120f88bc-d7ec-43a0-8503-726fb3ca3947","Type":"ContainerStarted","Data":"4560443a15934d45d7babf42767683e1db4bdf3d9520ffa7d8c0190c5abc3e55"}
Nov 01 14:16:35.127837 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:35.127787    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"38e27d7e3a4b8b3b59f6de6ab62a43ca","Type":"ContainerStarted","Data":"dfcea310dcec4be5e83789e17f7d04be0c9d1d8dd73f842bb4100f9fafadfe1b"}
Nov 01 14:16:35.227118 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:35.227039    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g44v5" podStartSLOduration=3.034242996 podStartE2EDuration="8.227012836s" totalImagesPullingTime="5.19276984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:35.226512802 +0000 UTC m=+8.658046077" watchObservedRunningTime="2026-11-01 14:16:35.227012836 +0000 UTC m=+8.658546086"
Nov 01 14:16:35.227537 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:35.227499    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal" podStartSLOduration=6.227485166 podStartE2EDuration="6.227485166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:34.444844191 +0000 UTC m=+7.876377442" watchObservedRunningTime="2026-11-01 14:16:35.227485166 +0000 UTC m=+8.659018417"
Nov 01 14:16:35.331367 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:35.331276    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal" podStartSLOduration=6.331261355 podStartE2EDuration="6.331261355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:35.331077874 +0000 UTC m=+8.762611126" watchObservedRunningTime="2026-11-01 14:16:35.331261355 +0000 UTC m=+8.762794603"
Nov 01 14:16:36.133617 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:36.133536    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2nh9b" event={"ID":"14c45109-a849-4e82-9c13-c5d1af56806e","Type":"ContainerStarted","Data":"36ced48ed949cf1edfa2a0bfbfc4efec5545f0eb3b8aff3d6f046823f2488dce"}
Nov 01 14:16:36.750179 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:36.750145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:36.750342 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:36.750321    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:36.750463 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:36.750435    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret podName:87bcc491-1e33-442b-b230-0cb7ae30f544 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:44.750376616 +0000 UTC m=+18.181909855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret") pod "global-pull-secret-syncer-z9bf5" (UID: "87bcc491-1e33-442b-b230-0cb7ae30f544") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:36.851349 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:36.850901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:36.851349 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:36.850945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:36.851349 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:36.851039    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:36.851349 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:36.851084    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs podName:f9f644ed-0d59-4762-af3b-45eefe8edc78 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:44.851070613 +0000 UTC m=+18.282603854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs") pod "network-metrics-daemon-9c6pv" (UID: "f9f644ed-0d59-4762-af3b-45eefe8edc78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:36.851349 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:36.851156    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:36.851349 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:36.851170    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:36.851349 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:36.851179    2575 projected.go:196] Error preparing data for projected volume kube-api-access-xxhn8 for pod openshift-network-diagnostics/network-check-target-thrgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:36.851349 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:36.851205    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8 podName:ac04470f-39e4-48f5-8d71-364e1d95f644 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:44.851197346 +0000 UTC m=+18.282730587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xxhn8" (UniqueName: "kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8") pod "network-check-target-thrgt" (UID: "ac04470f-39e4-48f5-8d71-364e1d95f644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:37.039004 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:37.034668    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:37.039004 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:37.034774    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:37.039004 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:37.034846    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:37.039004 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:37.034971    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:37.039004 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:37.035050    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:37.039004 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:37.035132    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:37.137848 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:37.137759    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f" event={"ID":"4681a549-d1f9-4bed-82c9-21a214c885b6","Type":"ContainerStarted","Data":"35147d365f88437b5bcbe14a792e2eec1d8fd061743f4f5acd7e242b71cc4e25"}
Nov 01 14:16:38.142674 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:38.141816    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wn9x6" event={"ID":"2f945e0d-6eb2-4c2f-ac70-e37fffe05749","Type":"ContainerStarted","Data":"81a97d2ace4226d499c219a242c741a89369ccfddb66ec9fcf94ed0454ff69f4"}
Nov 01 14:16:38.248251 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:38.248098    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wn9x6" podStartSLOduration=2.641197768 podStartE2EDuration="11.248078952s" totalImagesPullingTime="8.606881184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:38.248077655 +0000 UTC m=+11.679610907" watchObservedRunningTime="2026-11-01 14:16:38.248078952 +0000 UTC m=+11.679612198"
Nov 01 14:16:38.249876 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:38.249592    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2nh9b" podStartSLOduration=4.85611322 podStartE2EDuration="11.249577718s" totalImagesPullingTime="6.393464498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:36.237416154 +0000 UTC m=+9.668949404" watchObservedRunningTime="2026-11-01 14:16:38.249577718 +0000 UTC m=+11.681110968"
Nov 01 14:16:38.765510 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:38.765467    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 14:16:39.033733 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.033535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:39.033991 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:39.033838    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:39.033991 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.033564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:39.033991 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:39.033956    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:39.033991 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.033584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:39.034261 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:39.034016    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:39.044156 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.044060    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T14:16:38.76548528Z","UUID":"730cc7f5-f4a4-4e78-ab8a-3a6c1b379a4e","Handler":null,"Name":"","Endpoint":""}
Nov 01 14:16:39.045718 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.045699    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 14:16:39.045831 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.045746    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 14:16:39.147591 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.147549    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tqbqm" event={"ID":"57920aa2-346d-4bc2-99a0-6e92f382fa30","Type":"ContainerStarted","Data":"bf0c876d5ac17290bf3c133fd8dbfc099c479404c481f94becbc19ad8daf879f"}
Nov 01 14:16:39.150024 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.149992    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f" event={"ID":"4681a549-d1f9-4bed-82c9-21a214c885b6","Type":"ContainerStarted","Data":"b720ffce03470f1f4c5cd83be42f39616bb0a3299e39e4dc9751d3bc5b632f4f"}
Nov 01 14:16:39.153054 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.153021    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerStarted","Data":"bae1714c666c3ccd5de43360fabdc69b38c3f87ef3541581967b4323219e537d"}
Nov 01 14:16:39.251598 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.251498    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tqbqm" podStartSLOduration=3.097137164 podStartE2EDuration="12.251479245s" totalImagesPullingTime="9.154342081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:39.250781696 +0000 UTC m=+12.682314946" watchObservedRunningTime="2026-11-01 14:16:39.251479245 +0000 UTC m=+12.683012507"
Nov 01 14:16:39.344562 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.344530    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:39.345258 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:39.345209    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:40.157273 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:40.157198    2575 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="bae1714c666c3ccd5de43360fabdc69b38c3f87ef3541581967b4323219e537d" exitCode=0
Nov 01 14:16:40.157747 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:40.157537    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerDied","Data":"bae1714c666c3ccd5de43360fabdc69b38c3f87ef3541581967b4323219e537d"}
Nov 01 14:16:40.160410 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:40.160362    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qfz2f" event={"ID":"881f3467-ef5c-4bcc-b730-5f07644265b6","Type":"ContainerStarted","Data":"270a96a26322153b91dceec376632643b5120c0ad171e096de733f0ec8748ff3"}
Nov 01 14:16:41.033049 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:41.033020    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:41.033219 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:41.033057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:41.033219 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:41.033057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:41.033219 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:41.033193    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:41.033219 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:41.033174    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:41.033373 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:41.033309    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:41.163894 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:41.163854    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f" event={"ID":"4681a549-d1f9-4bed-82c9-21a214c885b6","Type":"ContainerStarted","Data":"ac759928c4ef44ab3e82f288aebdcc5acedfadc6b7ab86f7e58041efd6aaaa99"}
Nov 01 14:16:41.165570 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:41.165541    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4slbg" event={"ID":"1a11f89e-8c0d-4412-999f-b73cd5d20e2d","Type":"ContainerStarted","Data":"b059d44bdd4e2c4053455ceae98f2c7e7e986417ebc2b7d63fb2246f300aecba"}
Nov 01 14:16:41.275838 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:41.275780    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qfz2f" podStartSLOduration=5.257553946 podStartE2EDuration="14.275760797s" totalImagesPullingTime="9.018206851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:40.368899035 +0000 UTC m=+13.800432288" watchObservedRunningTime="2026-11-01 14:16:41.275760797 +0000 UTC m=+14.707294037"
Nov 01 14:16:41.277001 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:41.276955    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f" podStartSLOduration=3.718323682 podStartE2EDuration="14.276939595s" totalImagesPullingTime="10.558615913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:41.275935349 +0000 UTC m=+14.707468598" watchObservedRunningTime="2026-11-01 14:16:41.276939595 +0000 UTC m=+14.708472846"
Nov 01 14:16:42.171068 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:42.171037    2575 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="370b212775d68ee9e8ba924c16b3a6e03b42ea4b8b913ef16b0ce0ee820b694a" exitCode=0
Nov 01 14:16:42.171664 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:42.171117    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerDied","Data":"370b212775d68ee9e8ba924c16b3a6e03b42ea4b8b913ef16b0ce0ee820b694a"}
Nov 01 14:16:42.371433 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:42.371364    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4slbg" podStartSLOduration=4.494174815 podStartE2EDuration="15.371351707s" totalImagesPullingTime="10.877176892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:41.383765519 +0000 UTC m=+14.815298769" watchObservedRunningTime="2026-11-01 14:16:42.371351707 +0000 UTC m=+15.802884955"
Nov 01 14:16:43.033104 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:43.033069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:43.033270 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:43.033138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:43.033270 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:43.033156    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:43.033341 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:43.033265    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:43.033374 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:43.033341    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:43.033469 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:43.033450    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:44.178825 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:44.178610    2575 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="906521fe79472a64fe5c3a3a95fcd2c6ade9eb8d0b60a511970361c992e051b3" exitCode=0
Nov 01 14:16:44.178825 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:44.178699    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerDied","Data":"906521fe79472a64fe5c3a3a95fcd2c6ade9eb8d0b60a511970361c992e051b3"}
Nov 01 14:16:44.812939 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:44.812915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:44.813035 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:44.813023    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:44.813077 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:44.813069    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret podName:87bcc491-1e33-442b-b230-0cb7ae30f544 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:00.813057193 +0000 UTC m=+34.244590433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret") pod "global-pull-secret-syncer-z9bf5" (UID: "87bcc491-1e33-442b-b230-0cb7ae30f544") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:44.913649 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:44.913484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:44.913649 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:44.913635    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:44.913829 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:44.913662    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:44.913829 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:44.913672    2575 projected.go:196] Error preparing data for projected volume kube-api-access-xxhn8 for pod openshift-network-diagnostics/network-check-target-thrgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:44.913829 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:44.913674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:44.913829 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:44.913712    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8 podName:ac04470f-39e4-48f5-8d71-364e1d95f644 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:00.913698549 +0000 UTC m=+34.345231776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xxhn8" (UniqueName: "kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8") pod "network-check-target-thrgt" (UID: "ac04470f-39e4-48f5-8d71-364e1d95f644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:44.913829 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:44.913762    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:44.913829 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:44.913807    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs podName:f9f644ed-0d59-4762-af3b-45eefe8edc78 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:00.913795189 +0000 UTC m=+34.345328435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs") pod "network-metrics-daemon-9c6pv" (UID: "f9f644ed-0d59-4762-af3b-45eefe8edc78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:45.034010 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.033948    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:45.036806 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.033954    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:45.036806 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:45.034477    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:45.036806 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.034789    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:45.036806 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:45.034927    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:45.036806 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:45.035469    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:45.183993 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.183966    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:16:45.184453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.184365    2575 generic.go:356] "Generic (PLEG): container finished" podID="08c55c12-d679-4874-9e6f-789e7829641c" containerID="1b7c7098dae29ee5c83051f954c0bebc2ca1babb91621b5fe43d234a71764be9" exitCode=1
Nov 01 14:16:45.184453 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.184422    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"ad9875b65e97a1cd51505cd921c19187eefb035db74694df39f1a2ed63b94311"}
Nov 01 14:16:45.184576 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.184502    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"6e4ab7052cff344afd6ffb4fcbb79f4c03429fc0b46ede29e78507924fcd8389"}
Nov 01 14:16:45.184576 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.184538    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"175dda44e64fc9a8234539ac7a9cb7bfe5ac5bfb4e84f368404d27a997c01a78"}
Nov 01 14:16:45.184576 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.184566    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"109bada5b8be13ce37769fc19c2a4719e930eb1010f59b4f22ab32148cae3d98"}
Nov 01 14:16:45.184660 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.184592    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerDied","Data":"1b7c7098dae29ee5c83051f954c0bebc2ca1babb91621b5fe43d234a71764be9"}
Nov 01 14:16:45.184660 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:45.184621    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"8e6beae3da382a56d662ce2fb4665f1eb91f5eca7fc9ec3850742817fa2a533c"}
Nov 01 14:16:47.033356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:47.033328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:47.033356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:47.033356    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:47.033859 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:47.033335    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:47.033859 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:47.033441    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:47.033859 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:47.033498    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:47.033859 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:47.033570    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:47.190596 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:47.190517    2575 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="17327d36580352c5731dcc82bf7567dfd8043322c9ee6ab5542b5fefea8c2eef" exitCode=0
Nov 01 14:16:47.190745 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:47.190594    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerDied","Data":"17327d36580352c5731dcc82bf7567dfd8043322c9ee6ab5542b5fefea8c2eef"}
Nov 01 14:16:47.196263 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:47.193977    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:16:47.196665 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:47.196637    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"964d05f17f8a78ca4c589da6015d46d7c30fb23c3e05aba62123cb1d19d62711"}
Nov 01 14:16:48.201974 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:48.201939    2575 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="09e6771ea2c45422c1a6e254ea665b74b6e14f2fc03f5af6e7852c9a40696d63" exitCode=0
Nov 01 14:16:48.202488 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:48.202004    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerDied","Data":"09e6771ea2c45422c1a6e254ea665b74b6e14f2fc03f5af6e7852c9a40696d63"}
Nov 01 14:16:49.033845 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.033675    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:49.033845 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.033700    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:49.033845 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.033788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:49.033845 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:49.033795    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:49.034123 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:49.033869    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:49.034123 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:49.033944    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:49.208356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.208048    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86fr6" event={"ID":"746ff76c-dd84-49d4-b178-c687185af620","Type":"ContainerStarted","Data":"42244b6d20685514d7de33d52e9d4de14351c5faf1d491564a5a108ae24679b4"}
Nov 01 14:16:49.211278 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.211258    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:16:49.211613 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.211594    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"5563f7dbcdf14fe0651e26a07d502b77ca82ef7a5a4cd901f88847f56156a3da"}
Nov 01 14:16:49.211855 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.211833    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:49.211933 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.211909    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:49.211991 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.211957    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:49.212035 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.212002    2575 scope.go:122] "RemoveContainer" containerID="1b7c7098dae29ee5c83051f954c0bebc2ca1babb91621b5fe43d234a71764be9"
Nov 01 14:16:49.228693 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.228661    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:49.228936 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.228919    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:16:49.355474 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:49.355422    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-86fr6" podStartSLOduration=9.170653596 podStartE2EDuration="22.355410307s" totalImagesPullingTime="13.184756711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:49.355074038 +0000 UTC m=+22.786607287" watchObservedRunningTime="2026-11-01 14:16:49.355410307 +0000 UTC m=+22.786943556"
Nov 01 14:16:50.218484 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:50.218456    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:16:50.218943 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:50.218918    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" event={"ID":"08c55c12-d679-4874-9e6f-789e7829641c","Type":"ContainerStarted","Data":"87288ba1f89327abe2b1d464683d337d68652f2ed7523cad3113ceb6bbd8b15e"}
Nov 01 14:16:50.468605 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:50.468520    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s" podStartSLOduration=8.446548224 podStartE2EDuration="23.468505456s" totalImagesPullingTime="15.021957232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:50.452497244 +0000 UTC m=+23.884030493" watchObservedRunningTime="2026-11-01 14:16:50.468505456 +0000 UTC m=+23.900038705"
Nov 01 14:16:51.033111 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.033073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:51.033362 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:51.033181    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:51.033362 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.033200    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:51.033362 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.033087    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:51.033362 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:51.033304    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:51.033531 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:51.033411    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:51.682960 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.682923    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-thrgt"]
Nov 01 14:16:51.683542 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.683085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:51.683542 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:51.683188    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:51.688165 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.688144    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z9bf5"]
Nov 01 14:16:51.688279 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.688246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:51.688361 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:51.688340    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:51.694848 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.694825    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9c6pv"]
Nov 01 14:16:51.694934 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.694923    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:51.695024 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:51.695008    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:51.771635 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.771599    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:51.771785 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.771746    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 14:16:51.772120 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:51.772096    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:16:53.033646 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:53.033620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:53.034085 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:53.033620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:53.034085 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:53.033732    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:53.034085 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:53.033769    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:53.034085 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:53.033871    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:53.034085 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:53.033948    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:54.334180 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:54.334154    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tqbqm_57920aa2-346d-4bc2-99a0-6e92f382fa30/dns-node-resolver/0.log"
Nov 01 14:16:54.765700 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:54.765634    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2nh9b_14c45109-a849-4e82-9c13-c5d1af56806e/node-ca/0.log"
Nov 01 14:16:55.033343 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:55.033264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:55.033343 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:55.033278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:55.033564 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:55.033378    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:55.033564 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:55.033448    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:55.033564 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:55.033492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:55.033699 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:55.033569    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:57.033730 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:57.033701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:57.034076 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:57.033760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:57.034076 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:57.033770    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:57.034076 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:57.033847    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:16:57.034076 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:57.033919    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:57.034076 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:57.034001    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:59.033582 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:59.033402    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:16:59.033985 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:59.033433    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:16:59.033985 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:59.033654    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:16:59.033985 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:16:59.033485    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:16:59.033985 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:59.033755    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:16:59.033985 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:16:59.033819    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:00.842707 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:00.842675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:00.843190 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:00.842788    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:17:00.843190 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:00.842834    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret podName:87bcc491-1e33-442b-b230-0cb7ae30f544 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:32.842821766 +0000 UTC m=+66.274354993 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret") pod "global-pull-secret-syncer-z9bf5" (UID: "87bcc491-1e33-442b-b230-0cb7ae30f544") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:17:00.943371 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:00.943342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:00.943550 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:00.943418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:00.943550 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:00.943491    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:17:00.943550 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:00.943516    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:17:00.943550 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:00.943531    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:17:00.943550 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:00.943540    2575 projected.go:196] Error preparing data for projected volume kube-api-access-xxhn8 for pod openshift-network-diagnostics/network-check-target-thrgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:17:00.943550 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:00.943543    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs podName:f9f644ed-0d59-4762-af3b-45eefe8edc78 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:32.943529983 +0000 UTC m=+66.375063215 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs") pod "network-metrics-daemon-9c6pv" (UID: "f9f644ed-0d59-4762-af3b-45eefe8edc78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:17:00.943761 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:00.943573    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8 podName:ac04470f-39e4-48f5-8d71-364e1d95f644 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:32.943562512 +0000 UTC m=+66.375095740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xxhn8" (UniqueName: "kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8") pod "network-check-target-thrgt" (UID: "ac04470f-39e4-48f5-8d71-364e1d95f644") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:17:01.033421 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:01.033377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:01.033538 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:01.033377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:01.033538 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:01.033500    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:01.033538 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:01.033490    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:17:01.033655 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:01.033562    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:17:01.033655 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:01.033612    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:03.033473 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:03.033429    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:03.033473 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:03.033464    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:03.033908 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:03.033441    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:03.033908 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:03.033583    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:17:03.033908 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:03.033654    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:03.033908 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:03.033725    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:17:05.033194 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:05.033161    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:05.033634 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:05.033168    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:05.033634 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:05.033282    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:05.033634 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:05.033274    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:05.033634 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:05.033372    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:17:05.033634 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:05.033463    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:17:07.035895 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:07.035865    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:07.036274 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:07.035940    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:07.036274 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:07.036024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:07.036274 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:07.036075    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:07.036274 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:07.036087    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:17:07.036274 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:07.036155    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:17:09.033824 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:09.033801    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:09.034078 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:09.033808    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:09.034078 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:09.033899    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:17:09.034078 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:09.033916    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:09.034078 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:09.033991    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:17:09.034205 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:09.034078    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:11.033275 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:11.033240    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:11.033275 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:11.033273    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:11.033710 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:11.033332    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:17:11.033710 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:11.033429    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:11.033710 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:11.033504    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:11.033710 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:11.033580    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:17:13.033134 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:13.033105    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:13.033134 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:13.033121    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:13.033685 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:13.033121    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:13.033685 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:13.033223    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:13.033685 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:13.033278    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:17:13.033685 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:13.033341    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:17:15.033356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.033328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:15.033356 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.033345    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:15.033753 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:15.033467    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thrgt" podUID="ac04470f-39e4-48f5-8d71-364e1d95f644"
Nov 01 14:17:15.033753 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:15.033521    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9bf5" podUID="87bcc491-1e33-442b-b230-0cb7ae30f544"
Nov 01 14:17:15.033753 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.033555    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:15.033753 ip-10-0-135-142 kubenswrapper[2575]: E1101 14:17:15.033629    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9c6pv" podUID="f9f644ed-0d59-4762-af3b-45eefe8edc78"
Nov 01 14:17:15.837976 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.837897    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeReady"
Nov 01 14:17:15.838112 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.838018    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 14:17:15.942023 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.941994    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xxjqs"]
Nov 01 14:17:15.943923 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.943904    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8rbwt"]
Nov 01 14:17:15.944038 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.944022    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:15.945531 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.945515    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zq7mp"]
Nov 01 14:17:15.945647 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.945632    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:15.947072 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.947053    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:15.978874 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.978855    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxjqs"]
Nov 01 14:17:15.992561 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.992539    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 14:17:15.992732 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.992715    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjzzz\""
Nov 01 14:17:15.992811 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.992798    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 14:17:15.993191 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.993176    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 14:17:15.993435 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.993421    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 14:17:15.997649 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:15.997628    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rbwt"]
Nov 01 14:17:16.009079 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.009058    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zq7mp"]
Nov 01 14:17:16.054696 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f52851f6-a87d-4eee-81ea-e4882afab858-metrics-tls\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.054696 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054697    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4789a977-cfa4-479f-9e1e-bd5467d96e49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrdw\" (UniqueName: \"kubernetes.io/projected/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-api-access-wnrdw\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc007ccc-0f98-48f3-a43e-bec86a8d4523-cert\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4789a977-cfa4-479f-9e1e-bd5467d96e49-data-volume\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f52851f6-a87d-4eee-81ea-e4882afab858-config-volume\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7957\" (UniqueName: \"kubernetes.io/projected/cc007ccc-0f98-48f3-a43e-bec86a8d4523-kube-api-access-h7957\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054901    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4789a977-cfa4-479f-9e1e-bd5467d96e49-crio-socket\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.054993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52851f6-a87d-4eee-81ea-e4882afab858-tmp-dir\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.055009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7cd\" (UniqueName: \"kubernetes.io/projected/f52851f6-a87d-4eee-81ea-e4882afab858-kube-api-access-md7cd\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.055063 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.055024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.092982 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.092934    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 14:17:16.093047 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.093006    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 14:17:16.093047 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.093018    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 14:17:16.093143 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.093125    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 14:17:16.093214 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.093203    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8xv6q\""
Nov 01 14:17:16.093277 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.093232    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x2zjc\""
Nov 01 14:17:16.093333 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.093288    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 14:17:16.155934 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.155912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52851f6-a87d-4eee-81ea-e4882afab858-tmp-dir\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.156046 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.155936    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7cd\" (UniqueName: \"kubernetes.io/projected/f52851f6-a87d-4eee-81ea-e4882afab858-kube-api-access-md7cd\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.156046 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.155952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.156046 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.155983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f52851f6-a87d-4eee-81ea-e4882afab858-metrics-tls\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.156046 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4789a977-cfa4-479f-9e1e-bd5467d96e49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.156236 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrdw\" (UniqueName: \"kubernetes.io/projected/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-api-access-wnrdw\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.156236 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc007ccc-0f98-48f3-a43e-bec86a8d4523-cert\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:16.156236 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156113    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4789a977-cfa4-479f-9e1e-bd5467d96e49-data-volume\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.156236 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f52851f6-a87d-4eee-81ea-e4882afab858-config-volume\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.156472 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156454    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52851f6-a87d-4eee-81ea-e4882afab858-tmp-dir\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.156542 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7957\" (UniqueName: \"kubernetes.io/projected/cc007ccc-0f98-48f3-a43e-bec86a8d4523-kube-api-access-h7957\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:16.156579 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4789a977-cfa4-479f-9e1e-bd5467d96e49-crio-socket\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.156704 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156686    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.156760 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156724    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f52851f6-a87d-4eee-81ea-e4882afab858-config-volume\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.156760 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156744    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4789a977-cfa4-479f-9e1e-bd5467d96e49-crio-socket\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.156934 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.156913    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4789a977-cfa4-479f-9e1e-bd5467d96e49-data-volume\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.159888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.159862    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4789a977-cfa4-479f-9e1e-bd5467d96e49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.159969 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.159889    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc007ccc-0f98-48f3-a43e-bec86a8d4523-cert\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:16.159969 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.159893    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f52851f6-a87d-4eee-81ea-e4882afab858-metrics-tls\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.208485 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.208460    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7cd\" (UniqueName: \"kubernetes.io/projected/f52851f6-a87d-4eee-81ea-e4882afab858-kube-api-access-md7cd\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.208569 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.208508    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrdw\" (UniqueName: \"kubernetes.io/projected/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-api-access-wnrdw\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.208615 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.208603    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7957\" (UniqueName: \"kubernetes.io/projected/cc007ccc-0f98-48f3-a43e-bec86a8d4523-kube-api-access-h7957\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:16.255995 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.255976    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:16.261677 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.261663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:16.268503 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.268480    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:16.634684 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.634654    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxjqs"]
Nov 01 14:17:16.638176 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:16.638152    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52851f6_a87d_4eee_81ea_e4882afab858.slice/crio-08d9b304a4c238f9deda926125d77a4e902030983df67e28ab866af14a2232e8 WatchSource:0}: Error finding container 08d9b304a4c238f9deda926125d77a4e902030983df67e28ab866af14a2232e8: Status 404 returned error can't find the container with id 08d9b304a4c238f9deda926125d77a4e902030983df67e28ab866af14a2232e8
Nov 01 14:17:16.681308 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.681288    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rbwt"]
Nov 01 14:17:16.683982 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:16.683799    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zq7mp"]
Nov 01 14:17:16.687901 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:16.687877    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4789a977_cfa4_479f_9e1e_bd5467d96e49.slice/crio-f3f6a235a4f4119cc40bf28b5e45ec717c7608adec5416b388644865a37032a6 WatchSource:0}: Error finding container f3f6a235a4f4119cc40bf28b5e45ec717c7608adec5416b388644865a37032a6: Status 404 returned error can't find the container with id f3f6a235a4f4119cc40bf28b5e45ec717c7608adec5416b388644865a37032a6
Nov 01 14:17:17.035552 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.035474    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:17.035552 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.035512    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:17.035770 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.035644    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:17.084223 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.084194    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 14:17:17.084572 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.084360    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ftvdm\""
Nov 01 14:17:17.084572 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.084431    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2nj9\""
Nov 01 14:17:17.084572 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.084480    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 14:17:17.084572 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.084506    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 14:17:17.084572 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.084553    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 14:17:17.084718 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.084635    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mpq57\""
Nov 01 14:17:17.274953 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.274922    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zq7mp" event={"ID":"4789a977-cfa4-479f-9e1e-bd5467d96e49","Type":"ContainerStarted","Data":"2b6b69be970a90acbdc8c84d61ecd18a4b26746d7b1fbde4047734aa7297f3ab"}
Nov 01 14:17:17.275082 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.275005    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zq7mp" event={"ID":"4789a977-cfa4-479f-9e1e-bd5467d96e49","Type":"ContainerStarted","Data":"f3f6a235a4f4119cc40bf28b5e45ec717c7608adec5416b388644865a37032a6"}
Nov 01 14:17:17.276048 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.276024    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rbwt" event={"ID":"cc007ccc-0f98-48f3-a43e-bec86a8d4523","Type":"ContainerStarted","Data":"3ec4ba57ec0ce46660c7a597b7d6d96c697a57a2daa209816036b4dbc055b2f1"}
Nov 01 14:17:17.277087 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:17.277064    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxjqs" event={"ID":"f52851f6-a87d-4eee-81ea-e4882afab858","Type":"ContainerStarted","Data":"08d9b304a4c238f9deda926125d77a4e902030983df67e28ab866af14a2232e8"}
Nov 01 14:17:18.283880 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:18.283849    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zq7mp" event={"ID":"4789a977-cfa4-479f-9e1e-bd5467d96e49","Type":"ContainerStarted","Data":"67a185966ad4df907a7f6c9d891e3a84427b0a08baf38e826f7579091e609e65"}
Nov 01 14:17:19.289177 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:19.289076    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxjqs" event={"ID":"f52851f6-a87d-4eee-81ea-e4882afab858","Type":"ContainerStarted","Data":"0b503f6867cb11c84dc19863470882431bb0375fadd14ae09c331f10390bb5d0"}
Nov 01 14:17:19.289177 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:19.289129    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:19.289177 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:19.289152    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxjqs" event={"ID":"f52851f6-a87d-4eee-81ea-e4882afab858","Type":"ContainerStarted","Data":"e746337ea17517472bbc6d32d43d3b25f9ff6db9e5c956fa05a560df39561543"}
Nov 01 14:17:19.395736 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:19.395113    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xxjqs" podStartSLOduration=1.943536146 podStartE2EDuration="4.395092724s" totalImagesPullingTime="2.451556578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:19.393927414 +0000 UTC m=+52.825460684" watchObservedRunningTime="2026-11-01 14:17:19.395092724 +0000 UTC m=+52.826625974"
Nov 01 14:17:20.297297 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:20.297264    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rbwt" event={"ID":"cc007ccc-0f98-48f3-a43e-bec86a8d4523","Type":"ContainerStarted","Data":"a9d5adb3ec46d5803660dfc2f7004b9276cf088b2a17555c9abd6bd025847fc6"}
Nov 01 14:17:20.403464 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:20.403421    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8rbwt" podStartSLOduration=2.122704488 podStartE2EDuration="5.403408065s" totalImagesPullingTime="3.280703577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:20.402655305 +0000 UTC m=+53.834188572" watchObservedRunningTime="2026-11-01 14:17:20.403408065 +0000 UTC m=+53.834941311"
Nov 01 14:17:21.231919 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:21.231892    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:21.300978 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:21.300951    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zq7mp" event={"ID":"4789a977-cfa4-479f-9e1e-bd5467d96e49","Type":"ContainerStarted","Data":"71bd1ffe53a44c5ca9ae85302926f509f477110c3614a35f2159a6ae59e721c0"}
Nov 01 14:17:21.542435 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:21.542346    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zq7mp" podStartSLOduration=3.079861439 podStartE2EDuration="6.542332465s" totalImagesPullingTime="3.462471026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:17:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:21.54149672 +0000 UTC m=+54.973029980" watchObservedRunningTime="2026-11-01 14:17:21.542332465 +0000 UTC m=+54.973865744"
Nov 01 14:17:25.481056 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.481024    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"]
Nov 01 14:17:25.484144 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.484130    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.526294 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.526261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlswd\" (UniqueName: \"kubernetes.io/projected/9a370ac2-b58d-42cf-91b8-2754d966dc04-kube-api-access-vlswd\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.526442 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.526333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a370ac2-b58d-42cf-91b8-2754d966dc04-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.526507 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.526444    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.526507 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.526484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.579265 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.579239    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 14:17:25.579473 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.579457    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 14:17:25.579546 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.579514    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 14:17:25.579546 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.579531    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 14:17:25.579944 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.579929    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 14:17:25.580624 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.580610    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"]
Nov 01 14:17:25.581151 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.581134    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"]
Nov 01 14:17:25.584130 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.584113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.626769 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.626747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.626926 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.626806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a370ac2-b58d-42cf-91b8-2754d966dc04-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.626926 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.626864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.627041 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.626938    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.627041 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.626971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.627041 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.627020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/607bc669-1dea-4cec-9a9b-5d726e2ea979-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.627175 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.627091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.627175 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.627110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2h6\" (UniqueName: \"kubernetes.io/projected/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-api-access-8s2h6\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.627175 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.627138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlswd\" (UniqueName: \"kubernetes.io/projected/9a370ac2-b58d-42cf-91b8-2754d966dc04-kube-api-access-vlswd\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.627307 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.627181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.627517 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.627500    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vvzfx"]
Nov 01 14:17:25.627600 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.627582    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a370ac2-b58d-42cf-91b8-2754d966dc04-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.629491 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.629466    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.629578 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.629562    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.630927 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.630914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.633475 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.633460    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 14:17:25.633475 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.633465    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mxbff\""
Nov 01 14:17:25.633610 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.633491    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 14:17:25.633610 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.633516    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h4vvk\""
Nov 01 14:17:25.637055 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.637039    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 14:17:25.673501 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.673480    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"]
Nov 01 14:17:25.680078 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.680059    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlswd\" (UniqueName: \"kubernetes.io/projected/9a370ac2-b58d-42cf-91b8-2754d966dc04-kube-api-access-vlswd\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.728236 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-root\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.728316 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-metrics-client-ca\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.728316 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/607bc669-1dea-4cec-9a9b-5d726e2ea979-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.728435 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-textfile\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.728435 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.728534 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728460    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.728534 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-wtmp\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.728611 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728538    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-sys\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.728611 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.728611 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2h6\" (UniqueName: \"kubernetes.io/projected/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-api-access-8s2h6\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.728611 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728583    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/607bc669-1dea-4cec-9a9b-5d726e2ea979-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.728611 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.728772 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-tls\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.728873 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.728912 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.728962 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.728916    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47ttc\" (UniqueName: \"kubernetes.io/projected/afc89564-0e45-485c-9c9f-07284715d978-kube-api-access-47ttc\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.729090 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.729070    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.729438 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.729417    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.730705 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.730686    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.730950 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.730932    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.733832 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.733782    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 14:17:25.734010 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.733986    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5wqpw\""
Nov 01 14:17:25.734133 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.734062    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 14:17:25.737216 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.737197    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 14:17:25.785377 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.785357    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2h6\" (UniqueName: \"kubernetes.io/projected/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-api-access-8s2h6\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.799318 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.799300    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:25.829340 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-sys\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829476 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-tls\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829476 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ttc\" (UniqueName: \"kubernetes.io/projected/afc89564-0e45-485c-9c9f-07284715d978-kube-api-access-47ttc\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829476 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829431    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-sys\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829476 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-root\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829613 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829494    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-root\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829613 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829534    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-metrics-client-ca\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829613 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-textfile\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829613 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829794 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829794 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-wtmp\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.829894 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829834    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-wtmp\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.830503 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.829940    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-textfile\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.830503 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.830117    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-metrics-client-ca\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.830503 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.830461    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.831540 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.831520    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-tls\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.831725 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.831698    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.892983 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.892929    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ttc\" (UniqueName: \"kubernetes.io/projected/afc89564-0e45-485c-9c9f-07284715d978-kube-api-access-47ttc\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:25.898783 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.898760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:25.952665 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:25.952639    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:26.205464 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:26.205429    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"]
Nov 01 14:17:26.243090 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:26.243067    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a370ac2_b58d_42cf_91b8_2754d966dc04.slice/crio-6d77e6e0888c0261e39f571c809c2d6de7008980614af18dea3c876ca28bca12 WatchSource:0}: Error finding container 6d77e6e0888c0261e39f571c809c2d6de7008980614af18dea3c876ca28bca12: Status 404 returned error can't find the container with id 6d77e6e0888c0261e39f571c809c2d6de7008980614af18dea3c876ca28bca12
Nov 01 14:17:26.259975 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:26.259952    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"]
Nov 01 14:17:26.270307 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:26.270286    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607bc669_1dea_4cec_9a9b_5d726e2ea979.slice/crio-58a635a42747eba9577ce4bc1310877aef746775e947ecaf5055f6b736a85a2f WatchSource:0}: Error finding container 58a635a42747eba9577ce4bc1310877aef746775e947ecaf5055f6b736a85a2f: Status 404 returned error can't find the container with id 58a635a42747eba9577ce4bc1310877aef746775e947ecaf5055f6b736a85a2f
Nov 01 14:17:26.314089 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:26.314060    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9" event={"ID":"607bc669-1dea-4cec-9a9b-5d726e2ea979","Type":"ContainerStarted","Data":"58a635a42747eba9577ce4bc1310877aef746775e947ecaf5055f6b736a85a2f"}
Nov 01 14:17:26.315219 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:26.315191    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvzfx" event={"ID":"afc89564-0e45-485c-9c9f-07284715d978","Type":"ContainerStarted","Data":"746d7557b0c89ecb8785f44344c01f3a4f91e35b6877fb1b9eff1d79a1d141e7"}
Nov 01 14:17:26.316522 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:26.316502    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5" event={"ID":"9a370ac2-b58d-42cf-91b8-2754d966dc04","Type":"ContainerStarted","Data":"bedd5babee4cb5b887ffa7bf7d692895a93b51b9b27f3e84ac86e0c21afb7a39"}
Nov 01 14:17:26.316585 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:26.316545    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5" event={"ID":"9a370ac2-b58d-42cf-91b8-2754d966dc04","Type":"ContainerStarted","Data":"6d77e6e0888c0261e39f571c809c2d6de7008980614af18dea3c876ca28bca12"}
Nov 01 14:17:27.321511 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:27.321479    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5" event={"ID":"9a370ac2-b58d-42cf-91b8-2754d966dc04","Type":"ContainerStarted","Data":"d8cf6d16e62ec527a5a7c7f6f556a8549cf5c58aac0278105eabb5e10ddef733"}
Nov 01 14:17:28.152652 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.152618    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"]
Nov 01 14:17:28.156155 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.156131    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.207438 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.207413    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 14:17:28.207438 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.207415    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 14:17:28.207670 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.207652    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 14:17:28.249048 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.249130 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tfh7\" (UniqueName: \"kubernetes.io/projected/a61c1192-df73-4a89-88d3-9ebb9d890827-kube-api-access-8tfh7\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.249173 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.249216 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.249216 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.249354 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.249440 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249379    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61c1192-df73-4a89-88d3-9ebb9d890827-tmp\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.249440 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-grpc-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.249537 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.249517    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61c1192-df73-4a89-88d3-9ebb9d890827-metrics-client-ca\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.253421 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.253363    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 14:17:28.253421 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.253371    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 14:17:28.253524 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.253427    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k29xh\""
Nov 01 14:17:28.253524 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.253479    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c07asod0mkf0b\""
Nov 01 14:17:28.254513 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.254496    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"]
Nov 01 14:17:28.325623 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.325599    2575 generic.go:356] "Generic (PLEG): container finished" podID="afc89564-0e45-485c-9c9f-07284715d978" containerID="bbd2df8f9eb3b9d1323b9fd996a8da31b040f1876950d94b43180ce13c513649" exitCode=0
Nov 01 14:17:28.325919 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.325676    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvzfx" event={"ID":"afc89564-0e45-485c-9c9f-07284715d978","Type":"ContainerDied","Data":"bbd2df8f9eb3b9d1323b9fd996a8da31b040f1876950d94b43180ce13c513649"}
Nov 01 14:17:28.327841 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.327820    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5" event={"ID":"9a370ac2-b58d-42cf-91b8-2754d966dc04","Type":"ContainerStarted","Data":"9ddbbd770b1cf5c4ba0f73d10d0e49324d018d8123dad485e1dd7d2bd24c98a0"}
Nov 01 14:17:28.329362 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.329338    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9" event={"ID":"607bc669-1dea-4cec-9a9b-5d726e2ea979","Type":"ContainerStarted","Data":"9670f7650c2fa2dbffc627a4fc8e73b15c2bd6394198c230fd2ce2ef260aac20"}
Nov 01 14:17:28.350580 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.350686 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61c1192-df73-4a89-88d3-9ebb9d890827-tmp\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.350686 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-grpc-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.350845 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350673    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61c1192-df73-4a89-88d3-9ebb9d890827-metrics-client-ca\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.350845 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350735    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.350845 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfh7\" (UniqueName: \"kubernetes.io/projected/a61c1192-df73-4a89-88d3-9ebb9d890827-kube-api-access-8tfh7\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.350845 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.351041 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.351041 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.350883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.352660 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.352636    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61c1192-df73-4a89-88d3-9ebb9d890827-tmp\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.353314 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.353212    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61c1192-df73-4a89-88d3-9ebb9d890827-metrics-client-ca\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.353557 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.353533    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.353996 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.353970    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.354206 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.354183    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.355630 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.355601    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.355812 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.355797    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-grpc-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.356143 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.356127    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.405054 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.404930    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tfh7\" (UniqueName: \"kubernetes.io/projected/a61c1192-df73-4a89-88d3-9ebb9d890827-kube-api-access-8tfh7\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.479419 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.479372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:28.539731 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.539615    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5" podStartSLOduration=1.8911216830000002 podStartE2EDuration="3.539600693s" totalImagesPullingTime="1.64847901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:28.538930624 +0000 UTC m=+61.970463871" watchObservedRunningTime="2026-11-01 14:17:28.539600693 +0000 UTC m=+61.971133942"
Nov 01 14:17:28.889448 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:28.889414    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"]
Nov 01 14:17:29.299440 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:29.299346    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:29.337788 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:29.337750    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvzfx" event={"ID":"afc89564-0e45-485c-9c9f-07284715d978","Type":"ContainerStarted","Data":"d749649e6513699c76ed47a4dc4d5a1abd230bddf8a0f46d6b637ce8c82462c2"}
Nov 01 14:17:29.338200 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:29.337832    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvzfx" event={"ID":"afc89564-0e45-485c-9c9f-07284715d978","Type":"ContainerStarted","Data":"6824bb9cc43322728a9f5f9f5822a9fdef53d1af7cad941054d47e291091a5e9"}
Nov 01 14:17:29.338990 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:29.338962    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv" event={"ID":"a61c1192-df73-4a89-88d3-9ebb9d890827","Type":"ContainerStarted","Data":"b35c2d8f660f0a8d0d659e58c525a7db3481bf5272ed0975ce1d38ba2b1ac0a4"}
Nov 01 14:17:29.340980 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:29.340959    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9" event={"ID":"607bc669-1dea-4cec-9a9b-5d726e2ea979","Type":"ContainerStarted","Data":"ab3b7e973fb3a5585bc1f85783e0fb22c0defc4fc202e900f095ade6f8ebeb09"}
Nov 01 14:17:29.341053 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:29.340999    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9" event={"ID":"607bc669-1dea-4cec-9a9b-5d726e2ea979","Type":"ContainerStarted","Data":"974fafa0118cc0474b61be7e5b01c84e52ed5654118be8cb8a78f95935fe8d71"}
Nov 01 14:17:29.512065 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:29.512020    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vvzfx" podStartSLOduration=2.842499147 podStartE2EDuration="4.512006825s" totalImagesPullingTime="1.58985028s" totalInitContainerRuntime="79.657398ms" isStatefulPod=true podCreationTimestamp="2026-11-01 14:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:29.511464228 +0000 UTC m=+62.942997475" watchObservedRunningTime="2026-11-01 14:17:29.512006825 +0000 UTC m=+62.943540073"
Nov 01 14:17:29.618144 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:29.618098    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9" podStartSLOduration=2.65268757 podStartE2EDuration="4.61808323s" totalImagesPullingTime="1.96539566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:29.616884686 +0000 UTC m=+63.048417934" watchObservedRunningTime="2026-11-01 14:17:29.61808323 +0000 UTC m=+63.049616478"
Nov 01 14:17:30.383332 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.383303    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-klphz"]
Nov 01 14:17:30.386679 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.386658    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:30.436876 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.436847    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 14:17:30.437003 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.436914    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xrrrs\""
Nov 01 14:17:30.437364 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.437337    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-klphz"]
Nov 01 14:17:30.468872 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.468837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vq57\" (UniqueName: \"kubernetes.io/projected/3585c408-fcee-40eb-9184-220f1f51a1ab-kube-api-access-2vq57\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:30.468988 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.468900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3585c408-fcee-40eb-9184-220f1f51a1ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:30.570340 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.570308    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vq57\" (UniqueName: \"kubernetes.io/projected/3585c408-fcee-40eb-9184-220f1f51a1ab-kube-api-access-2vq57\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:30.570512 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.570356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3585c408-fcee-40eb-9184-220f1f51a1ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:30.573107 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.573076    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3585c408-fcee-40eb-9184-220f1f51a1ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:30.623252 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.623217    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vq57\" (UniqueName: \"kubernetes.io/projected/3585c408-fcee-40eb-9184-220f1f51a1ab-kube-api-access-2vq57\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:30.700155 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:30.700083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:31.069621 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:31.069588    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-klphz"]
Nov 01 14:17:31.072666 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:31.072636    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3585c408_fcee_40eb_9184_220f1f51a1ab.slice/crio-d5f7e15ab0234995f54d907c56b04d7d2c9c66e02c9e626561ea63d33a7828b1 WatchSource:0}: Error finding container d5f7e15ab0234995f54d907c56b04d7d2c9c66e02c9e626561ea63d33a7828b1: Status 404 returned error can't find the container with id d5f7e15ab0234995f54d907c56b04d7d2c9c66e02c9e626561ea63d33a7828b1
Nov 01 14:17:31.348506 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:31.348431    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz" event={"ID":"3585c408-fcee-40eb-9184-220f1f51a1ab","Type":"ContainerStarted","Data":"d5f7e15ab0234995f54d907c56b04d7d2c9c66e02c9e626561ea63d33a7828b1"}
Nov 01 14:17:32.353997 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:32.353960    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv" event={"ID":"a61c1192-df73-4a89-88d3-9ebb9d890827","Type":"ContainerStarted","Data":"c5dd8084252024ce66813bdc3389a73e3291c197a256f51062d71c83878216b2"}
Nov 01 14:17:32.354534 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:32.354037    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv" event={"ID":"a61c1192-df73-4a89-88d3-9ebb9d890827","Type":"ContainerStarted","Data":"32a0093eb8d56539670bd2df3ac8af2f86103403765d5396d7108ba36682fee9"}
Nov 01 14:17:32.354534 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:32.354064    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv" event={"ID":"a61c1192-df73-4a89-88d3-9ebb9d890827","Type":"ContainerStarted","Data":"d4891ebc46ca3e52242ecd97dd647e29fb3ce6cfb5375ccf7555377419cefaf1"}
Nov 01 14:17:32.890104 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:32.890072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:32.937651 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:32.937624    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 14:17:32.942811 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:32.942784    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:32.990610 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:32.990585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:32.990732 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:32.990631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:33.004427 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.004405    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mpq57\""
Nov 01 14:17:33.007150 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.007135    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:33.038121 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.038094    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 14:17:33.038241 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.038228    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 14:17:33.054353 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.054289    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:33.090947 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.090911    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 14:17:33.096888 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.096861    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:33.295078 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.295015    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2nj9\""
Nov 01 14:17:33.298163 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.298139    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:33.299846 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.299825    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ftvdm\""
Nov 01 14:17:33.302813 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.302796    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:33.368117 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.368080    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz" event={"ID":"3585c408-fcee-40eb-9184-220f1f51a1ab","Type":"ContainerStarted","Data":"3c6bed46f8230ab05ca61adcae5f9e33fb8d666ed94b4e8f8ac606ab8f84d6a5"}
Nov 01 14:17:33.368485 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.368423    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:33.373343 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.373322    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:33.373626 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.373612    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z9bf5"]
Nov 01 14:17:33.389495 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:33.389474    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bcc491_1e33_442b_b230_0cb7ae30f544.slice/crio-3ac5ee3e6f51220e49ff20ce104e45b7014b334fd4e4c7af2bcc3c8869384c68 WatchSource:0}: Error finding container 3ac5ee3e6f51220e49ff20ce104e45b7014b334fd4e4c7af2bcc3c8869384c68: Status 404 returned error can't find the container with id 3ac5ee3e6f51220e49ff20ce104e45b7014b334fd4e4c7af2bcc3c8869384c68
Nov 01 14:17:33.477421 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.477363    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz" podStartSLOduration=1.579844364 podStartE2EDuration="3.477350017s" totalImagesPullingTime="1.897505653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:33.476192557 +0000 UTC m=+66.907725808" watchObservedRunningTime="2026-11-01 14:17:33.477350017 +0000 UTC m=+66.908883266"
Nov 01 14:17:33.660436 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.660405    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9c6pv"]
Nov 01 14:17:33.663378 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:33.663345    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f644ed_0d59_4762_af3b_45eefe8edc78.slice/crio-d7789f08555f18f8394014d8cf819929149ad7942889b5ee2d965f1ff2cda8a1 WatchSource:0}: Error finding container d7789f08555f18f8394014d8cf819929149ad7942889b5ee2d965f1ff2cda8a1: Status 404 returned error can't find the container with id d7789f08555f18f8394014d8cf819929149ad7942889b5ee2d965f1ff2cda8a1
Nov 01 14:17:33.663676 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:33.663650    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-thrgt"]
Nov 01 14:17:33.666612 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:33.666588    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac04470f_39e4_48f5_8d71_364e1d95f644.slice/crio-733517d2ea0af15813b0c11c4ebe59cc8b5c04c40b40de10bc20aa8d39948bd3 WatchSource:0}: Error finding container 733517d2ea0af15813b0c11c4ebe59cc8b5c04c40b40de10bc20aa8d39948bd3: Status 404 returned error can't find the container with id 733517d2ea0af15813b0c11c4ebe59cc8b5c04c40b40de10bc20aa8d39948bd3
Nov 01 14:17:34.373710 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.373674    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv" event={"ID":"a61c1192-df73-4a89-88d3-9ebb9d890827","Type":"ContainerStarted","Data":"ce159bf3323077f459943b3b232f5cd161c682693d81cb3a219a8f25397d8a51"}
Nov 01 14:17:34.374173 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.373754    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv" event={"ID":"a61c1192-df73-4a89-88d3-9ebb9d890827","Type":"ContainerStarted","Data":"a8a6e9528005dc5e9e4dffcaf04eb9b8a388aeb8364b237235e8c3a1545696c2"}
Nov 01 14:17:34.374173 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.373792    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv" event={"ID":"a61c1192-df73-4a89-88d3-9ebb9d890827","Type":"ContainerStarted","Data":"c008ee7afa97d97ea3eae6f9042b0a285edd9b5389dc356e54e528de09778f26"}
Nov 01 14:17:34.374173 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.373857    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:34.375012 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.374975    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9c6pv" event={"ID":"f9f644ed-0d59-4762-af3b-45eefe8edc78","Type":"ContainerStarted","Data":"d7789f08555f18f8394014d8cf819929149ad7942889b5ee2d965f1ff2cda8a1"}
Nov 01 14:17:34.376130 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.376112    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-thrgt" event={"ID":"ac04470f-39e4-48f5-8d71-364e1d95f644","Type":"ContainerStarted","Data":"733517d2ea0af15813b0c11c4ebe59cc8b5c04c40b40de10bc20aa8d39948bd3"}
Nov 01 14:17:34.377225 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.377205    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z9bf5" event={"ID":"87bcc491-1e33-442b-b230-0cb7ae30f544","Type":"ContainerStarted","Data":"3ac5ee3e6f51220e49ff20ce104e45b7014b334fd4e4c7af2bcc3c8869384c68"}
Nov 01 14:17:34.488782 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.488734    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv" podStartSLOduration=2.204481014 podStartE2EDuration="6.488719503s" totalImagesPullingTime="4.284238489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:34.487543142 +0000 UTC m=+67.919076411" watchObservedRunningTime="2026-11-01 14:17:34.488719503 +0000 UTC m=+67.920252751"
Nov 01 14:17:34.735297 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.735230    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-sggrb"]
Nov 01 14:17:34.738569 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.738547    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:34.791097 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.791055    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 14:17:34.791291 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.791093    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 14:17:34.793960 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.793935    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sggrb"]
Nov 01 14:17:34.794198 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.794158    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-9qhws\""
Nov 01 14:17:34.803815 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.803794    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/34ab3520-5128-4753-81e2-463deb5298d3-kube-api-access-w8zw6\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:34.803927 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.803835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ab3520-5128-4753-81e2-463deb5298d3-tmp\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:34.904951 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.904410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/34ab3520-5128-4753-81e2-463deb5298d3-kube-api-access-w8zw6\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:34.904951 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.904488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ab3520-5128-4753-81e2-463deb5298d3-tmp\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:34.904951 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.904907    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ab3520-5128-4753-81e2-463deb5298d3-tmp\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:34.959196 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:34.959161    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/34ab3520-5128-4753-81e2-463deb5298d3-kube-api-access-w8zw6\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:35.053800 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:35.053724    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:35.533626 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:35.533591    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sggrb"]
Nov 01 14:17:35.573018 ip-10-0-135-142 kubenswrapper[2575]: W1101 14:17:35.572991    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ab3520_5128_4753_81e2_463deb5298d3.slice/crio-3e21298e483e75306f0d1c5aa19135bceda8c7e65d19c81f62e99751060a3738 WatchSource:0}: Error finding container 3e21298e483e75306f0d1c5aa19135bceda8c7e65d19c81f62e99751060a3738: Status 404 returned error can't find the container with id 3e21298e483e75306f0d1c5aa19135bceda8c7e65d19c81f62e99751060a3738
Nov 01 14:17:36.386535 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:36.386501    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9c6pv" event={"ID":"f9f644ed-0d59-4762-af3b-45eefe8edc78","Type":"ContainerStarted","Data":"23cc086a3f77d8c7bf3bf4e1d729185fde0e1af6aff71b80835878c3752c31ab"}
Nov 01 14:17:36.386703 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:36.386565    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9c6pv" event={"ID":"f9f644ed-0d59-4762-af3b-45eefe8edc78","Type":"ContainerStarted","Data":"f72e5ce301acbe4d53962237fdbda2c5a46eba031947e001da28675e7f6c8cd9"}
Nov 01 14:17:36.387931 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:36.387899    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sggrb" event={"ID":"34ab3520-5128-4753-81e2-463deb5298d3","Type":"ContainerStarted","Data":"3e21298e483e75306f0d1c5aa19135bceda8c7e65d19c81f62e99751060a3738"}
Nov 01 14:17:36.495150 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:36.495087    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9c6pv" podStartSLOduration=67.666798513 podStartE2EDuration="1m9.495067833s" totalImagesPullingTime="1.82826932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:36.493437664 +0000 UTC m=+69.924970914" watchObservedRunningTime="2026-11-01 14:17:36.495067833 +0000 UTC m=+69.926601085"
Nov 01 14:17:38.398429 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:38.398377    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-thrgt" event={"ID":"ac04470f-39e4-48f5-8d71-364e1d95f644","Type":"ContainerStarted","Data":"d7aa3291c29efa18904f767a5763088079c2f052913379175669943fe8ca8a15"}
Nov 01 14:17:38.398854 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:38.398550    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:38.505374 ip-10-0-135-142 kubenswrapper[2575]: I1101 14:17:38.505310    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-thrgt" podStartSLOduration=67.405079998 podStartE2EDuration="1m11.505292361s" totalImagesPullingTime="4.100212363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:38.503847268 +0000 UTC m=+71.935380519" watchObservedRunningTime="2026-11-01 14:17:38.505292361 +0000 UTC m=+71.936825611"
Nov 01 14:17:39.777576 ip-10-0-135-142 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 14:17:39.791906 ip-10-0-135-142 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 14:17:39.792132 ip-10-0-135-142 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 14:17:39.792188 ip-10-0-135-142 systemd[1]: kubelet.service: Consumed 3.793s CPU time.
Nov 01 14:17:39.802963 ip-10-0-135-142 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 14:17:39.866473 ip-10-0-135-142 kubenswrapper[5423]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:17:39.866473 ip-10-0-135-142 kubenswrapper[5423]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 14:17:39.866473 ip-10-0-135-142 kubenswrapper[5423]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:17:39.866473 ip-10-0-135-142 kubenswrapper[5423]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:17:39.868622 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868536    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:17:39.868622 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868621    5423 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:17:39.868622 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868625    5423 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:17:39.868622 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868628    5423 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868632    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868637    5423 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868641    5423 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868644    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868647    5423 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868661    5423 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868665    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868668    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868670    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868673    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868676    5423 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868679    5423 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868682    5423 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868685    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868687    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868690    5423 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868692    5423 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868695    5423 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:17:39.868769 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868698    5423 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868700    5423 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868703    5423 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868707    5423 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868709    5423 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868712    5423 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868715    5423 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868718    5423 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868721    5423 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868723    5423 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868726    5423 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868729    5423 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868731    5423 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868734    5423 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868737    5423 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868739    5423 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868742    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868745    5423 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868748    5423 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:17:39.869234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868751    5423 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868754    5423 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868757    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868759    5423 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868762    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868765    5423 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868767    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868770    5423 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868772    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868775    5423 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868778    5423 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868781    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868784    5423 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868786    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868789    5423 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868792    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868794    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868798    5423 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868802    5423 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868805    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:17:39.869718 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868808    5423 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868812    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868814    5423 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868817    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868820    5423 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868822    5423 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868825    5423 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868829    5423 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868832    5423 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868835    5423 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868838    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868841    5423 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868844    5423 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868846    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868849    5423 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868852    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868854    5423 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868857    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868859    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868862    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:17:39.870214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868864    5423 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868867    5423 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868870    5423 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868872    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868875    5423 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868877    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868880    5423 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868883    5423 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868885    5423 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868888    5423 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868891    5423 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868893    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868896    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868899    5423 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868901    5423 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868904    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868910    5423 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868912    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868915    5423 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868918    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:17:39.870736 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868921    5423 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868924    5423 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868926    5423 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868929    5423 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.868937    5423 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869740    5423 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869748    5423 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869753    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869757    5423 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869760    5423 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869763    5423 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869766    5423 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869768    5423 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869771    5423 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869780    5423 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869783    5423 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869785    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869788    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869791    5423 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869793    5423 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:17:39.871264 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869796    5423 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869799    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869801    5423 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869804    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869807    5423 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869810    5423 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869812    5423 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869815    5423 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869822    5423 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869825    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869828    5423 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869831    5423 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869834    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869837    5423 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869865    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869869    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869871    5423 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869877    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869882    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:17:39.871816 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869886    5423 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869890    5423 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869894    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869899    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869902    5423 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869904    5423 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869907    5423 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869910    5423 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869913    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869916    5423 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869918    5423 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869921    5423 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869924    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869926    5423 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869929    5423 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869932    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869936    5423 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869941    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869945    5423 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:17:39.872271 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869949    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869952    5423 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869957    5423 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869962    5423 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869966    5423 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869971    5423 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869975    5423 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869978    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869980    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869983    5423 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869986    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869989    5423 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869992    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869995    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.869998    5423 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870000    5423 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870003    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870006    5423 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870009    5423 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:17:39.872776 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870011    5423 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870014    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870017    5423 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870019    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870022    5423 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870024    5423 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870027    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870030    5423 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870032    5423 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870036    5423 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870040    5423 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870045    5423 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870049    5423 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870052    5423 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870055    5423 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870060    5423 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870064    5423 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870070    5423 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870074    5423 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870078    5423 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:17:39.873242 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870081    5423 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870084    5423 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870086    5423 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870089    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870092    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870094    5423 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870098    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870101    5423 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870105    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870108    5423 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870110    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870114    5423 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870118    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870123    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870703    5423 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870711    5423 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870714    5423 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870716    5423 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870719    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870722    5423 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:17:39.873827 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870725    5423 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870728    5423 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870730    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870733    5423 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870736    5423 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870739    5423 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870742    5423 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870747    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870750    5423 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870753    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870756    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870758    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870761    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870764    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870768    5423 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870772    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870775    5423 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870778    5423 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870782    5423 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:17:39.874353 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870785    5423 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870788    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870790    5423 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870793    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870796    5423 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870798    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870801    5423 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870804    5423 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870807    5423 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870810    5423 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870812    5423 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870815    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870818    5423 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870821    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870825    5423 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870830    5423 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870833    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870837    5423 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870840    5423 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:17:39.874879 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870842    5423 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870846    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870848    5423 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870852    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870855    5423 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870858    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870860    5423 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870863    5423 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870866    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870868    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870871    5423 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870874    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870876    5423 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870879    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870882    5423 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870885    5423 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870887    5423 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870890    5423 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870893    5423 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870895    5423 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:17:39.875426 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870898    5423 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870900    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870903    5423 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870906    5423 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870908    5423 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870911    5423 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870913    5423 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870916    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870918    5423 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870921    5423 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870924    5423 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870927    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870929    5423 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870932    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870935    5423 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870940    5423 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870942    5423 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870945    5423 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870948    5423 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870950    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:17:39.876020 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870953    5423 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870956    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870958    5423 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870961    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870963    5423 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870966    5423 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870969    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870972    5423 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870974    5423 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870977    5423 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870980    5423 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870982    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870985    5423 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870988    5423 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870990    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.870993    5423 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871003    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871007    5423 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871009    5423 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871012    5423 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:17:39.876585 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871015    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871017    5423 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871106    5423 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871113    5423 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871119    5423 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871125    5423 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871129    5423 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871134    5423 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871139    5423 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871143    5423 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871146    5423 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871149    5423 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871153    5423 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871156    5423 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871160    5423 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871163    5423 flags.go:64] FLAG: --cgroup-root=""
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871166    5423 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871169    5423 flags.go:64] FLAG: --client-ca-file=""
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871172    5423 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871175    5423 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871180    5423 flags.go:64] FLAG: --cluster-domain=""
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871183    5423 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871186    5423 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871189    5423 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 14:17:39.877093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871193    5423 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871196    5423 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871200    5423 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871229    5423 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871233    5423 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871236    5423 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871241    5423 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871245    5423 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871250    5423 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871257    5423 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871263    5423 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871266    5423 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871269    5423 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871273    5423 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871276    5423 flags.go:64] FLAG: --enable-server="true"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871279    5423 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871284    5423 flags.go:64] FLAG: --event-burst="100"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871287    5423 flags.go:64] FLAG: --event-qps="50"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871291    5423 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871294    5423 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871297    5423 flags.go:64] FLAG: --eviction-hard=""
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871301    5423 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871304    5423 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871308    5423 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871311    5423 flags.go:64] FLAG: --eviction-soft=""
Nov 01 14:17:39.877698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871314    5423 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871318    5423 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871323    5423 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871329    5423 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871334    5423 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871338    5423 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871344    5423 flags.go:64] FLAG: --feature-gates=""
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871350    5423 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871354    5423 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871360    5423 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871364    5423 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871369    5423 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871374    5423 flags.go:64] FLAG: --help="false"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871379    5423 flags.go:64] FLAG: --hostname-override="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871399    5423 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871405    5423 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871412    5423 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871418    5423 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871423    5423 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871428    5423 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871433    5423 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871437    5423 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871442    5423 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 14:17:39.878314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871447    5423 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871452    5423 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871457    5423 flags.go:64] FLAG: --kube-reserved=""
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871462    5423 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871466    5423 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871471    5423 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871475    5423 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871480    5423 flags.go:64] FLAG: --lock-file=""
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871485    5423 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871489    5423 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871493    5423 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871499    5423 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871503    5423 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871507    5423 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871510    5423 flags.go:64] FLAG: --logging-format="text"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871515    5423 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871519    5423 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871522    5423 flags.go:64] FLAG: --manifest-url=""
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871525    5423 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871530    5423 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871533    5423 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871538    5423 flags.go:64] FLAG: --max-pods="110"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871541    5423 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871544    5423 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871547    5423 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 14:17:39.878880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871551    5423 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871554    5423 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871557    5423 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871561    5423 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871568    5423 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871571    5423 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871574    5423 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871578    5423 flags.go:64] FLAG: --pod-cidr=""
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871581    5423 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871584    5423 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871587    5423 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871590    5423 flags.go:64] FLAG: --port="10250"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871593    5423 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871596    5423 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-049690c63fd336593"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871600    5423 flags.go:64] FLAG: --qos-reserved=""
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871603    5423 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871606    5423 flags.go:64] FLAG: --register-node="true"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871609    5423 flags.go:64] FLAG: --register-with-taints=""
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871613    5423 flags.go:64] FLAG: --registry-burst="10"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871616    5423 flags.go:64] FLAG: --registry-qps="5"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871619    5423 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871623    5423 flags.go:64] FLAG: --reserved-memory=""
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871633    5423 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871637    5423 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871640    5423 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871643    5423 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 14:17:39.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871646    5423 flags.go:64] FLAG: --runonce="false"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871649    5423 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871652    5423 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871656    5423 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871659    5423 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871662    5423 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871666    5423 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871670    5423 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871673    5423 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871676    5423 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871679    5423 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871682    5423 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871685    5423 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871688    5423 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871692    5423 flags.go:64] FLAG: --system-cgroups=""
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871695    5423 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871700    5423 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871703    5423 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871706    5423 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871710    5423 flags.go:64] FLAG: --tls-min-version=""
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871713    5423 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871716    5423 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871719    5423 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871722    5423 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871725    5423 flags.go:64] FLAG: --v="2"
Nov 01 14:17:39.880703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871730    5423 flags.go:64] FLAG: --version="false"
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871734    5423 flags.go:64] FLAG: --vmodule=""
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871738    5423 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.871746    5423 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871889    5423 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871893    5423 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871897    5423 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871900    5423 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871903    5423 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871906    5423 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871909    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871912    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871915    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871919    5423 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871922    5423 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871925    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871928    5423 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871931    5423 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871934    5423 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871937    5423 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:17:39.881527 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871939    5423 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871942    5423 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871945    5423 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871948    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871950    5423 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871953    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871956    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871958    5423 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871961    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871964    5423 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871967    5423 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871969    5423 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871972    5423 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871975    5423 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871977    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871981    5423 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871984    5423 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871987    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871990    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871992    5423 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:17:39.882016 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871995    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.871998    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872001    5423 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872003    5423 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872006    5423 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872010    5423 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872012    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872015    5423 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872019    5423 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872023    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872026    5423 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872029    5423 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872031    5423 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872034    5423 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872037    5423 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872039    5423 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872042    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872045    5423 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872047    5423 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:17:39.882525 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872050    5423 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872053    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872055    5423 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872058    5423 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872061    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872064    5423 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872067    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872070    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872073    5423 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872075    5423 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872078    5423 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872082    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872084    5423 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872087    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872090    5423 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872092    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872095    5423 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872098    5423 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872102    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:17:39.883167 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872104    5423 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872107    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872110    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872112    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872115    5423 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872118    5423 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872120    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872123    5423 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872126    5423 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872128    5423 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872131    5423 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872135    5423 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872139    5423 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872142    5423 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872146    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872149    5423 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872153    5423 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872155    5423 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872158    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:17:39.883858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872161    5423 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872164    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872167    5423 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872170    5423 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872172    5423 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872176    5423 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872179    5423 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872181    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872184    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872186    5423 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872189    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872192    5423 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872196    5423 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.872224    5423 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872258    5423 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872263    5423 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:17:39.884410 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872267    5423 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872271    5423 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872276    5423 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872279    5423 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872282    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872286    5423 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872289    5423 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872292    5423 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872295    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872298    5423 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872301    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872304    5423 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872306    5423 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872309    5423 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872311    5423 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872314    5423 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872317    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872320    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872323    5423 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872325    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:17:39.884812 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872328    5423 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872331    5423 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872334    5423 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872337    5423 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872339    5423 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872342    5423 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872345    5423 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872348    5423 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872351    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872355    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872358    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872360    5423 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872363    5423 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872366    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872369    5423 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872372    5423 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872375    5423 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872377    5423 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872380    5423 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872398    5423 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:17:39.885293 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872401    5423 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872404    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872407    5423 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872410    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872412    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872415    5423 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872418    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872420    5423 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872423    5423 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872426    5423 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872430    5423 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872433    5423 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872435    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872438    5423 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872441    5423 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872444    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872447    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872450    5423 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872453    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872455    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:17:39.886081 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872458    5423 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872462    5423 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872465    5423 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872469    5423 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872472    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872474    5423 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872478    5423 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872482    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872486    5423 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872489    5423 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872491    5423 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872494    5423 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872497    5423 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872499    5423 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872502    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872505    5423 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872507    5423 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872510    5423 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872513    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:17:39.886672 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872516    5423 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872518    5423 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872521    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872525    5423 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872528    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872531    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872533    5423 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872536    5423 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872538    5423 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872542    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872545    5423 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872547    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872550    5423 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872552    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872556    5423 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872559    5423 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872562    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872565    5423 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872568    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:17:39.887188 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872570    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872573    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872576    5423 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872579    5423 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872582    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.872584    5423 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.876780    5423 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.876797    5423 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.876816    5423 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.876824    5423 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876884    5423 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876890    5423 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876893    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876896    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876899    5423 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876902    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876905    5423 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876907    5423 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876910    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876913    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:17:39.887679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876916    5423 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876918    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876921    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876924    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876926    5423 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876929    5423 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876932    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876934    5423 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876938    5423 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876941    5423 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876944    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876948    5423 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876950    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876953    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876956    5423 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876959    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876962    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876965    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876968    5423 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:17:39.888197 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876976    5423 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876979    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876982    5423 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876984    5423 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876987    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876990    5423 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876993    5423 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876995    5423 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.876998    5423 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877001    5423 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877003    5423 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877006    5423 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877009    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877011    5423 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877014    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877017    5423 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877019    5423 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877022    5423 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877025    5423 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:17:39.888761 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877027    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877030    5423 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877033    5423 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877036    5423 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877039    5423 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877042    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877045    5423 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877048    5423 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877051    5423 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877053    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877056    5423 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877059    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877062    5423 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877064    5423 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877072    5423 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877077    5423 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877081    5423 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877085    5423 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877088    5423 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877091    5423 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:17:39.889222 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877095    5423 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877098    5423 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877101    5423 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877104    5423 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877107    5423 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877110    5423 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877113    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877115    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877118    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877121    5423 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877124    5423 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877126    5423 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877129    5423 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877132    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877135    5423 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877138    5423 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877141    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877144    5423 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877146    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:17:39.889860 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877149    5423 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877152    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877154    5423 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877157    5423 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877160    5423 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877163    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877165    5423 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877168    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877177    5423 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877180    5423 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877183    5423 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877185    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877188    5423 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877191    5423 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877193    5423 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877197    5423 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877217    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877222    5423 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:17:39.890346 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877225    5423 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.877255    5423 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877465    5423 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877471    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877475    5423 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877478    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877480    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877483    5423 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877486    5423 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877489    5423 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877491    5423 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877494    5423 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877497    5423 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877499    5423 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877502    5423 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877505    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:17:39.890804 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877507    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877517    5423 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877520    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877523    5423 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877525    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877528    5423 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877530    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877539    5423 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877542    5423 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877544    5423 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877547    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877550    5423 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877552    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877555    5423 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877557    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877560    5423 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877563    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877565    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877568    5423 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:17:39.891208 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877571    5423 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877573    5423 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877576    5423 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877578    5423 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877581    5423 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877584    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877587    5423 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877589    5423 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877592    5423 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877594    5423 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877597    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877600    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877602    5423 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877605    5423 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877608    5423 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877610    5423 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877613    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877616    5423 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877618    5423 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877621    5423 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:17:39.891749 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877623    5423 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877631    5423 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877634    5423 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877637    5423 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877639    5423 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877642    5423 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877646    5423 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877650    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877654    5423 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877657    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877660    5423 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877663    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877666    5423 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877668    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877671    5423 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877673    5423 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877676    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877679    5423 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877681    5423 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:17:39.892224 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877684    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877687    5423 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877689    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877692    5423 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877695    5423 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877697    5423 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877700    5423 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877703    5423 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877705    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877708    5423 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877711    5423 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877714    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877717    5423 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877719    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877722    5423 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877730    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877733    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877735    5423 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877738    5423 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:17:39.892747 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877740    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877743    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877745    5423 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877748    5423 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877751    5423 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877753    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877756    5423 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877759    5423 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877761    5423 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877764    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877766    5423 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877769    5423 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877772    5423 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877774    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877777    5423 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.877801    5423 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:17:39.893214 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877845    5423 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877849    5423 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877851    5423 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877854    5423 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877857    5423 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877859    5423 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877862    5423 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877865    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877867    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877870    5423 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877873    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877875    5423 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877878    5423 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877889    5423 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877892    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877895    5423 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877897    5423 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877900    5423 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877903    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877905    5423 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:17:39.893695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877908    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877911    5423 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877914    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877916    5423 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877919    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877922    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877924    5423 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877927    5423 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877929    5423 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877932    5423 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877935    5423 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877938    5423 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877940    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877943    5423 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877946    5423 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877948    5423 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877951    5423 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877953    5423 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877956    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877959    5423 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:17:39.894347 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877961    5423 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877964    5423 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877967    5423 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877969    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877972    5423 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877974    5423 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877983    5423 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877986    5423 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877989    5423 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877991    5423 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877994    5423 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877996    5423 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.877999    5423 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878003    5423 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878007    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878011    5423 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878015    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878018    5423 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878021    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:17:39.895015 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878027    5423 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878031    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878033    5423 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878036    5423 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878039    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878041    5423 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878044    5423 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878047    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878050    5423 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878052    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878055    5423 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878057    5423 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878060    5423 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878063    5423 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878065    5423 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878068    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878070    5423 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878073    5423 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878075    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:17:39.895647 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878078    5423 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878086    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878089    5423 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878092    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878094    5423 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878097    5423 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878101    5423 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878104    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878106    5423 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878109    5423 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878112    5423 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878115    5423 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878117    5423 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878122    5423 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878125    5423 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878128    5423 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878131    5423 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878133    5423 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878136    5423 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:17:39.896106 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878139    5423 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878142    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878144    5423 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878147    5423 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878150    5423 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878153    5423 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878156    5423 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878159    5423 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:39.878161    5423 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.878220    5423 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.879551    5423 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.879627    5423 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.880192    5423 server.go:1014] "Starting client certificate rotation"
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.880504    5423 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.880535    5423 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 14:11:27 +0000 UTC" deadline="2028-05-21 17:08:40.049528781 +0000 UTC"
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.880638    5423 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13610h51m0.168914496s"
Nov 01 14:17:39.896617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.881148    5423 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 14:17:39.897039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.882513    5423 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 14:17:39.897039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.883787    5423 log.go:25] "Validated CRI v1 runtime API"
Nov 01 14:17:39.897039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.885648    5423 log.go:25] "Validated CRI v1 image API"
Nov 01 14:17:39.897039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.886401    5423 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 14:17:39.897039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.888361    5423 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9e00471d-7bdb-4dfa-8f33-835dc29ea08d:/dev/nvme0n1p4 fc362e37-508d-4f01-b43a-63b9653bbbb8:/dev/nvme0n1p3]
Nov 01 14:17:39.897211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.888381    5423 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08d9b304a4c238f9deda926125d77a4e902030983df67e28ab866af14a2232e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08d9b304a4c238f9deda926125d77a4e902030983df67e28ab866af14a2232e8/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bdc2b910c56d063d994085516c825f7efa30056144d57f46a0a7a969ceccbb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bdc2b910c56d063d994085516c825f7efa30056144d57f46a0a7a969ceccbb2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cfc1eb69dff98fe4a10b20fd76e3c5b961792435b474c2c02de16236cc7aeee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cfc1eb69dff98fe4a10b20fd76e3c5b961792435b474c2c02de16236cc7aeee/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37d0c0308a1a50f4e13dff6538ed02d2c3a5b704cdbfb225f8a678a78870f2b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37d0c0308a1a50f4e13dff6538ed02d2c3a5b704cdbfb225f8a678a78870f2b2/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ac5ee3e6f51220e49ff20ce104e45b7014b334fd4e4c7af2bcc3c8869384c68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ac5ee3e6f51220e49ff20ce104e45b7014b334fd4e4c7af2bcc3c8869384c68/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e21298e483e75306f0d1c5aa19135bceda8c7e65d19c81f62e99751060a3738/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e21298e483e75306f0d1c5aa19135bceda8c7e65d19c81f62e99751060a3738/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ec4ba57ec0ce46660c7a597b7d6d96c697a57a2daa209816036b4dbc055b2f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ec4ba57ec0ce46660c7a597b7d6d96c697a57a2daa209816036b4dbc055b2f1/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b0517da2462b1dce1e4b7a0f268fa4dddf46af677f7926f9062e8a330543399/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b0517da2462b1dce1e4b7a0f268fa4dddf46af677f7926f9062e8a330543399/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f95137e087a14eb0b1b186e49e0de878f9e03c618bfd3eafb035496c1b50fb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f95137e087a14eb0b1b186e49e0de878f9e03c618bfd3eafb035496c1b50fb0/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58a635a42747eba9577ce4bc1310877aef746775e947ecaf5055f6b736a85a2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58a635a42747eba9577ce4bc1310877aef746775e947ecaf5055f6b736a85a2f/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c201adaea417dcb703e8ca6f2cc5c2ceb78c2fb48a9b29f40921ac87df2020d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c201adaea417dcb703e8ca6f2cc5c2ceb78c2fb48a9b29f40921ac87df2020d/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d77e6e0888c0261e39f571c809c2d6de7008980614af18dea3c876ca28bca12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d77e6e0888c0261e39f571c809c2d6de7008980614af18dea3c876ca28bca12/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e9418e6ba607b400c5f12d970b7e59e9b54fc48a314098b0a4afcd1c9e95d7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e9418e6ba607b400c5f12d970b7e59e9b54fc48a314098b0a4afcd1c9e95d7e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/728bb1a9335a76ec4d258e7fae981ec3740a755af555a1b2fd2ce57b8c84d3d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/728bb1a9335a76ec4d258e7fae981ec3740a755af555a1b2fd2ce57b8c84d3d9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/733517d2ea0af15813b0c11c4ebe59cc8b5c04c40b40de10bc20aa8d39948bd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/733517d2ea0af15813b0c11c4ebe59cc8b5c04c40b40de10bc20aa8d39948bd3/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/746d7557b0c89ecb8785f44344c01f3a4f91e35b6877fb1b9eff1d79a1d141e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/746d7557b0c89ecb8785f44344c01f3a4f91e35b6877fb1b9eff1d79a1d141e7/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bf76f71fb7565eb8d5c13c0ee8f717273f4bae30fbcbd85fa429fcfc9882cce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bf76f71fb7565eb8d5c13c0ee8f717273f4bae30fbcbd85fa429fcfc9882cce/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b35c2d8f660f0a8d0d659e58c525a7db3481bf5272ed0975ce1d38ba2b1ac0a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b35c2d8f660f0a8d0d659e58c525a7db3481bf5272ed0975ce1d38ba2b1ac0a4/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7d6abce1d36443e2f8e24fb0f4f96fe2ca3867067b40a422373fecd8ad0ac1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7d6abce1d36443e2f8e24fb0f4f96fe2ca3867067b40a422373fecd8ad0ac1a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5f7e15ab0234995f54d907c56b04d7d2c9c66e02c9e626561ea63d33a7828b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5f7e15ab0234995f54d907c56b04d7d2c9c66e02c9e626561ea63d33a7828b1/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7789f08555f18f8394014d8cf819929149ad7942889b5ee2d965f1ff2cda8a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7789f08555f18f8394014d8cf819929149ad7942889b5ee2d965f1ff2cda8a1/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3f6a235a4f4119cc40bf28b5e45ec717c7608adec5416b388644865a37032a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3f6a235a4f4119cc40bf28b5e45ec717c7608adec5416b388644865a37032a6/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volumes/kubernetes.io~projected/kube-api-access-sfk4t:{mountpoint:/var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volumes/kubernetes.io~projected/kube-api-access-sfk4t major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/120f88bc-d7ec-43a0-8503-726fb3ca3947/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/120f88bc-d7ec-43a0-8503-726fb3ca3947/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14c45109-a849-4e82-9c13-c5d1af56806e/volumes/kubernetes.io~projected/kube-api-access-g4wrs:{mountpoint:/var/lib/kubelet/pods/14c45109-a849-4e82-9c13-c5d1af56806e/volumes/kubernetes.io~projected/kube-api-access-g4wrs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~projected/kube-api-access-8rlx6:{mountpoint:/var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~projected/kube-api-access-8rlx6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f945e0d-6eb2-4c2f-ac70-e37fffe05749/volumes/kubernetes.io~projected/kube-api-access-qj9zk:{mountpoint:/var/lib/kubelet/pods/2f945e0d-6eb2-4c2f-ac70-e37fffe05749/volumes/kubernetes.io~projected/kube-api-access-qj9zk major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34ab3520-5128-4753-81e2-463deb5298d3/volumes/kubernetes.io~projected/kube-api-access-w8zw6:{mountpoint:/var/lib/kubelet/pods/34ab3520-5128-4753-81e2-463deb5298d3/volumes/kubernetes.io~projected/kube-api-access-w8zw6 major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3585c408-fcee-40eb-9184-220f1f51a1ab/volumes/kubernetes.io~projected/kube-api-access-2vq57:{mountpoint:/var/lib/kubelet/pods/3585c408-fcee-40eb-9184-220f1f51a1ab/volumes/kubernetes.io~projected/kube-api-access-2vq57 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3585c408-fcee-40eb-9184-220f1f51a1ab/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/3585c408-fcee-40eb-9184-220f1f51a1ab/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4681a549-d1f9-4bed-82c9-21a214c885b6/volumes/kubernetes.io~projected/kube-api-access-n29kd:{mountpoint:/var/lib/kubelet/pods/4681a549-d1f9-4bed-82c9-21a214c885b6/volumes/kubernetes.io~projected/kube-api-access-n29kd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4789a977-cfa4-479f-9e1e-bd5467d96e49/volumes/kubernetes.io~projected/kube-api-access-wnrdw:{mountpoint:/var/lib/kubelet/pods/4789a977-cfa4-479f-9e1e-bd5467d96e49/volumes/kubernetes.io~projected/kube-api-access-wnrdw major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4789a977-cfa4-479f-9e1e-bd5467d96e49/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4789a977-cfa4-479f-9e1e-bd5467d96e49/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57920aa2-346d-4bc2-99a0-6e92f382fa30/volumes/kubernetes.io~projected/kube-api-access-8mfs4:{mountpoint:/var/lib/kubelet/pods/57920aa2-346d-4bc2-99a0-6e92f382fa30/volumes/kubernetes.io~projected/kube-api-access-8mfs4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~projected/kube-api-access-8s2h6:{mountpoint:/var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~projected/kube-api-access-8s2h6 major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/746ff76c-dd84-49d4-b178-c687185af620/volumes/kubernetes.io~projected/kube-api-access-b5ws4:{mountpoint:/var/lib/kubelet/pods/746ff76c-dd84-49d4-b178-c687185af620/volumes/kubernetes.io~projected/kube-api-access-b5ws4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87bcc491-1e33-442b-b230-0cb7ae30f544/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/87bcc491-1e33-442b-b230-0cb7ae30f544/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/881f3467-ef5c-4bcc-b730-5f07644265b6/volumes/kubernetes.io~projected/kube-api-access-899qc:{mountpoint:/var/lib/kubelet/pods/881f3467-ef5c-4bcc-b730-5f07644265b6/volumes/kubernetes.io~projected/kube-api-access-899qc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~projected/kube-api-access-vlswd:{mountpoint:/var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~projected/kube-api-access-vlswd major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~projected/kube-api-access-8tfh7:{mountpoint:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~projected/kube-api-access-8tfh7 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac04470f-39e4-48f5-8d71-364e1d95f644/volumes/kubernetes.io~projected/kube-api-access-xxhn8:{mountpoint:/var/lib/kubelet/pods/ac04470f-39e4-48f5-8d71-364e1d95f644/volumes/kubernetes.io~projected/kube-api-access-xxhn8 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~projected/kube-api-access-47ttc:{mountpoint:/var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~projected/kube-api-access-47ttc major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc007ccc-0f98-48f3-a43e-bec86a8d4523/volumes/kubernetes.io~projected/kube-api-access-h7957:{mountpoint:/var/lib/kubelet/pods/cc007ccc-0f98-48f3-a43e-bec86a8d4523/volumes/kubernetes.io~projected/kube-api-access-h7957 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc007ccc-0f98-48f3-a43e-bec86a8d4523/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cc007ccc-0f98-48f3-a43e-bec86a8d4523/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f52851f6-a87d-4eee-81ea-e4882afab858/volumes/kubernetes.io~projected/kube-api-access-md7cd:{mountpoint:/var/lib/kubelet/pods/f52851f6-a87d-4eee-81ea-e4882afab858/volumes/kubernetes.io~projected/kube-api-access-md7cd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f52851f6-a87d-4eee-81ea-e4882afab858/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f52851f6-a87d-4eee-81ea-e4882afab858/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9f644ed-0d59-4762-af3b-45eefe8edc78/volumes/kubernetes.io~projected/kube-api-access-7jlct:{mountpoint:/var/lib/kubelet/pods/f9f644ed-0d59-4762-af3b-45eefe8edc78/volumes/kubernetes.io~projected/kube-api-access-7jlct major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9f644ed-0d59-4762-af3b-45eefe8edc78/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f9f644ed-0d59-4762-af3b-45eefe8edc78/volumes/kubernetes.io~secret/metrics-certs major:0 minor:349 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cb8164f634b81ee1b17abe699d61d86a348eddc85b3b3bea4170624e034bed6c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/af956445228f503db4bb6cc1e4609b6d6de6dd52d18545ef5df28ea28b7cea54/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/38024930aeaca2fe2691a6bfe72c71c6a7aeebeb1cda4a6c43e3c211651f8934/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/abd7e99b3d0d9aeec90fe3c7ca4d43370b20c078c918488f59c408e289afec48/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d0abc7e79d9127151f8f2ef981be77fce7b98bb60099c3aa0b6799fc83856405/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d119968064510934185c67474376b32df117ca195c0bd170540b61c24d7fa8b3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6b2ba88bfa77e2cd627e052577a4fd20543779d887ac9950f8881fb6e2c91d7f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4cf3bdd5f7bdc1d2d6cea483971553515011edd89efe1551bead93376ee1e6af/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/bf370811b52a34a19b828b31d91c76d1555df232aab8d44863a90a6af4699afe/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f538273673e83a9fc5064cc4d2ee85a71e0e7c39c28e76ce0af3a179c0fdc1a1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/0a99023f6f1356dd257f7ca0ab5cd8c24873fd65c4f077e1751f0890be7cff9a/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c247eb1c082e4c042c0e19da670fe5c6e08112ccfd137617861be6d03614fbf0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/1c21f7223549b2f8f70d424b2edf8617cc9d7ff81fa394b4fd82c77c029c86df/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5c4c9982352c49cc0c2fb1c8371c0a51ce19e1d026570b50a5f700cd98ac59de/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c3e2e4ef269a5370be40305a8d5853a31706493d208b55c3c905a5444eed81ac/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2dfa5ba89681257160da9a9e35da08978a4cb47ba398dfde4c0a9b8662f720ef/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1dc7bdfb4e65a180cbfb844db3194035a23097549b29a544df0ddf386b81bea9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/86509fbcb905592b14279eb9faeda4baa151b41cb41a6dbeece95a616b972c40/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b14c8683affe6d8de88f6db89350589348b5ac0b2aed94c5f3613152e10599f0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cf64d46cc136989d46c6155f3cf684a42783cada3208e912377072d0358a237b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7666348914dc50969cfe48d3e6f5c4894030801741ed3bf3283fd5d8b3ed14f4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/851c4bb3dacc2278ca30cb1464c40710cf1c734caeaa4aa8cc95641f18c3538c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/57e50df8b38dea7aee4aa08dbd1e50710386661daf1118e557201a1df10b062e/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/cd4d7dda12771b0ca1f2507fb48eabc33a0f3b62fa6070589831b20223527b3d/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ff54b5199ffc08181b762b918817c6505f0c6d4b2c71e4bbeb6847930c96076c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/c5cbad1fc4e513911dad0139f22b9d88ca6a8f06045513f448b08ae42c0cba54/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/576f72461d5551548ab350effc38324934e4b0600eed0122a81af8567c403d66/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/57b9c1d672894ea1fea23d8df891a32157532c3228c1a8b17ae6a638a0ebf4a1/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/eadf7a3a60d77aa3b267ab02b1753b5fb4a415ccb6afb7cf0f4eab511c932b4c/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/463031540de1fa923b51d84426c4a98b999fb18a108638c1c36144a7df91404c/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/758418c2696afd37c0354174900743a957fa3fc50f9c0adc6ea0e7827c17f897/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/5922a3e1b02cde3df5a320668997f48a7a53b646e6b4f6060dead94fd9f06fd4/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/5a229dcde96c322bdc3001c0ce1144a335a4bfdce91cce4da3a21c1826396d13/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/8a4696dba0c1ebe108b530fef1cd1bcfedf7a66cac4ea698cfa012feb416e588/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/2eb5816483830c383f1c6043d29f8573079405de4d90a18f8b51be7617ace4be/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/c5f9f262d0725cc64728d45ecc20370d73a82d63f3785427453412886fbe9e30/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/9afab980740d088037cef73f713879e3efdb601b1a112476b9c457fda8f1e390/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/6e8e94d11acb391a114d4edafab08d71524526af1436af56200e61962439f69a/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/c19677bb67876cb7a873d63561ae23349ec54e6f96af3097885111dfd840d894/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/ce730c4b4e66752a7754e7bb87942022c71329b5a7dedd811e3ca2fc926c99ad/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/280216e36c1e2cff51c837cdf5ae0c206a3356e10aaa56c3735ab8c8acaf2b0b/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/6ad5b7b5ee25d5137ec4fae6589b9f337c95db666acab45696adf3c89ec0911b/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/a65dede8b1a54c875f97b8194d667a3f2f763ee784c94ea830638ee3ab176780/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/a3dcfc98201935d4ee166bd88996354cb10eff30ef44eb50b06cb720059cb14c/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/11a9c91c349479dae14fd76b4ad66b5af4b23a284e472d99f2d15c5dbf77a99b/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/1354a2fed7ec0e07c7e2987e1d46d3c86e86398a438394b8539f525d27f9e796/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/fcccb0eed4c0362ca3af87d99842a839f1135bc12d6432aa3a6d46611a8dc0c0/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/d8b877d0cd71a4983c6e41aa05d8756db5f4d6880eed68e296172e955b1690ec/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/bda5e4b286291c771ecfa6571a15778a29020acaf27a14c231d60a6d7b2727fc/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/8922a4c7bc3a353fa535dd6c03794d310b63d91f76894d817dd6a6eb9430b343/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/2c634d6a0905b046a9f5ebcb8ecd20171589bc67db0e7ef95d1de18d2678d974/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/3f761a3e25c9eb78d66b08e1b3849ce5248ad467353552e97c226565e3c41f1a/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/0e5ebeacf355f466da9de853ac27cab8a65b3d5dbe569169f6b3c04ef74893ac/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/9018d989e56db962458b7d28ad7fcedb8e3b5f7b49c8f2ad061c671fa09c601d/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/0e95942b3b9f330c37282a6cdff63477f107e755a8ef61d79368855040190f76/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/b196e7bfc03308e9b3804f1f1cf9d9ec5dd4f84dea9d84404c4858a02b76e349/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/9b9b86cef91f912928aeb32b32d7bbf9148ddcc877d9089f9da6509a0394bd86/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/7d81cd61fbaf885d3a3aa82cf1477886b5b43abd0545b67087cc6726506195be/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/f1f093ddacb3c60e2236d9d510f93cc09539a86227f61b2d89827a991f7e48a7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/aa7c0e32fd98298fb2472ea23bfe6681f796ac51f98174feb6a896f53744e17a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/67740a3246164967d6d9a846500e41f3cb49d6f5fa9066b3e50073853ffef18f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7cfa401c07331b76793b2054e3c359ffa0bffe2fd23a8b4c410592a3695c3666/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a088159c67c83341e5be2c77a28567079f3bc872b07ce8d726fa5e0a8749a041/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c3682393102908703f4449029dd890b12dfbbad0f9455b6a3db8f29074243707/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/53b5517fa2c51fb3bfabeb93e1daea73e49417b42a41c7e96caa8207a66de225/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b57572f446a4758a1bba80d2b1678f8ebf745b3dd4bc65349719f4479a695cb0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/764f1b3b09ef73e7b1bca98d1c7e63ea073f23b0b275abb82b799a457f3d9302/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/182360e4840ea1e5da40bbc1d9b6d8730a81dcf4c896d3e66f94a1765693625b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.895962    5423 manager.go:217] Machine: {Timestamp:2026-11-01 14:17:39.895496278 +0000 UTC m=+0.060981567 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101028 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc86f7c7b159b8da86698ee5d7280 SystemUUID:ec2fc86f-7c7b-159b-8da8-6698ee5d7280 BootID:57b27608-5a5a-41d1-944c-8b8f9696027c Filesystems:[{Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f52851f6-a87d-4eee-81ea-e4882afab858/volumes/kubernetes.io~projected/kube-api-access-md7cd DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~projected/kube-api-access-47ttc DeviceMajor:0 DeviceMinor:260 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4789a977-cfa4-479f-9e1e-bd5467d96e49/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4789a977-cfa4-479f-9e1e-bd5467d96e49/volumes/kubernetes.io~projected/kube-api-access-wnrdw DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bdc2b910c56d063d994085516c825f7efa30056144d57f46a0a7a969ceccbb2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/881f3467-ef5c-4bcc-b730-5f07644265b6/volumes/kubernetes.io~projected/kube-api-access-899qc DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~projected/kube-api-access-8s2h6 DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08d9b304a4c238f9deda926125d77a4e902030983df67e28ab866af14a2232e8/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34ab3520-5128-4753-81e2-463deb5298d3/volumes/kubernetes.io~projected/kube-api-access-w8zw6 DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f95137e087a14eb0b1b186e49e0de878f9e03c618bfd3eafb035496c1b50fb0/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~projected/kube-api-access-vlswd DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~projected/kube-api-access-8rlx6 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~projected/kube-api-access-8tfh7 DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc007ccc-0f98-48f3-a43e-bec86a8d4523/volumes/kubernetes.io~projected/kube-api-access-h7957 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7d6abce1d36443e2f8e24fb0f4f96fe2ca3867067b40a422373fecd8ad0ac1a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bf76f71fb7565eb8d5c13c0ee8f717273f4bae30fbcbd85fa429fcfc9882cce/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/746d7557b0c89ecb8785f44344c01f3a4f91e35b6877fb1b9eff1d79a1d141e7/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3f6a235a4f4119cc40bf28b5e45ec717c7608adec5416b388644865a37032a6/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58a635a42747eba9577ce4bc1310877aef746775e947ecaf5055f6b736a85a2f/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac04470f-39e4-48f5-8d71-364e1d95f644/volumes/kubernetes.io~projected/kube-api-access-xxhn8 DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volumes/kubernetes.io~projected/kube-api-access-sfk4t DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7789f08555f18f8394014d8cf819929149ad7942889b5ee2d965f1ff2cda8a1/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4681a549-d1f9-4bed-82c9-21a214c885b6/volumes/kubernetes.io~projected/kube-api-access-n29kd DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f52851f6-a87d-4eee-81ea-e4882afab858/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/607bc669-1dea-4cec-9a9b-5d726e2ea979/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/afc89564-0e45-485c-9c9f-07284715d978/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3585c408-fcee-40eb-9184-220f1f51a1ab/volumes/kubernetes.io~projected/kube-api-access-2vq57 DeviceMajor:0 DeviceMinor:325 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5f7e15ab0234995f54d907c56b04d7d2c9c66e02c9e626561ea63d33a7828b1/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87bcc491-1e33-442b-b230-0cb7ae30f544/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b35c2d8f660f0a8d0d659e58c525a7db3481bf5272ed0975ce1d38ba2b1ac0a4/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/57920aa2-346d-4bc2-99a0-6e92f382fa30/volumes/kubernetes.io~projected/kube-api-access-8mfs4 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/120f88bc-d7ec-43a0-8503-726fb3ca3947/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/14c45109-a849-4e82-9c13-c5d1af56806e/volumes/kubernetes.io~projected/kube-api-access-g4wrs DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ec4ba57ec0ce46660c7a597b7d6d96c697a57a2daa209816036b4dbc055b2f1/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/733517d2ea0af15813b0c11c4ebe59cc8b5c04c40b40de10bc20aa8d39948bd3/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/746ff76c-dd84-49d4-b178-c687185af620/volumes/kubernetes.io~projected/kube-api-access-b5ws4 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37d0c0308a1a50f4e13dff6538ed02d2c3a5b704cdbfb225f8a678a78870f2b2/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b0517da2462b1dce1e4b7a0f268fa4dddf46af677f7926f9062e8a330543399/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c201adaea417dcb703e8ca6f2cc5c2ceb78c2fb48a9b29f40921ac87df2020d/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:306 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a11f89e-8c0d-4412-999f-b73cd5d20e2d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3585c408-fcee-40eb-9184-220f1f51a1ab/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9f644ed-0d59-4762-af3b-45eefe8edc78/volumes/kubernetes.io~projected/kube-api-access-7jlct DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/728bb1a9335a76ec4d258e7fae981ec3740a755af555a1b2fd2ce57b8c84d3d9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e9418e6ba607b400c5f12d970b7e59e9b54fc48a314098b0a4afcd1c9e95d7e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cfc1eb69dff98fe4a10b20fd76e3c5b961792435b474c2c02de16236cc7aeee/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc007ccc-0f98-48f3-a43e-bec86a8d4523/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9f644ed-0d59-4762-af3b-45eefe8edc78/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:349 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e21298e483e75306f0d1c5aa19135bceda8c7e65d19c81f62e99751060a3738/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/08c55c12-d679-4874-9e6f-789e7829641c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a370ac2-b58d-42cf-91b8-2754d966dc04/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d77e6e0888c0261e39f571c809c2d6de7008980614af18dea3c876ca28bca12/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a61c1192-df73-4a89-88d3-9ebb9d890827/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ac5ee3e6f51220e49ff20ce104e45b7014b334fd4e4c7af2bcc3c8869384c68/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f945e0d-6eb2-4c2f-ac70-e37fffe05749/volumes/kubernetes.io~projected/kube-api-access-qj9zk DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08d9b304a4c238f MacAddress:be:70:ef:70:e7:cc Speed:10000 Mtu:8901} {Name:3ac5ee3e6f51220 MacAddress:c2:96:2a:8a:8c:b7 Speed:10000 Mtu:8901} {Name:3e21298e483e753 MacAddress:12:63:91:57:7f:dc Speed:10000 Mtu:8901} {Name:3ec4ba57ec0ce46 MacAddress:ea:96:7a:ea:46:ea Speed:10000 Mtu:8901} {Name:58a635a42747eba MacAddress:0a:a2:57:82:78:2f Speed:10000 Mtu:8901} {Name:6d77e6e0888c026 MacAddress:5e:3e:d1:2b:ee:72 Speed:10000 Mtu:8901} {Name:733517d2ea0af15 MacAddress:5e:e9:ff:83:dd:2a Speed:10000 Mtu:8901} {Name:b35c2d8f660f0a8 MacAddress:c2:e9:64:c0:16:cd Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c0:88:5f:c7 Speed:0 Mtu:9001} {Name:br-int MacAddress:66:ec:5e:45:8c:01 Speed:0 Mtu:8901} {Name:d5f7e15ab023499 MacAddress:ca:61:e4:f6:1b:1f Speed:10000 Mtu:8901} {Name:d7789f08555f18f MacAddress:ba:62:2c:68:c5:e2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c0:88:5f:c7 Speed:0 Mtu:9001} {Name:f3f6a235a4f4119 MacAddress:c6:d8:2b:ee:8d:47 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c6:02:7a:84:ed:3d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:31:c2:4e:8c:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896297    5423 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896339    5423 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896710    5423 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896730    5423 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-142.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896865    5423 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896874    5423 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896887    5423 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896905    5423 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.896979    5423 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.897126    5423 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.897231    5423 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.897263    5423 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.897274    5423 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.897283    5423 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.897291    5423 kubelet.go:405] "Adding apiserver pod source"
Nov 01 14:17:39.897521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.897298    5423 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 14:17:39.898837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.898819    5423 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 14:17:39.898837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.898837    5423 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 14:17:39.899558 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.899503    5423 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 14:17:39.912747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.912728    5423 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 14:17:39.912747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.912744    5423 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 14:17:39.912876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.912853    5423 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 14:17:39.913130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913115    5423 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 14:17:39.913187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913152    5423 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 14:17:39.913286 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913269    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 14:17:39.913286 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913290    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913300    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913317    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913326    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913332    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913338    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913344    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913352    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913358    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913377    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913400    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 14:17:39.913419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913424    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 14:17:39.913771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.913431    5423 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 14:17:39.917002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.916987    5423 server.go:1263] "Started kubelet"
Nov 01 14:17:39.917140 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.917079    5423 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 14:17:39.917201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.917127    5423 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 14:17:39.917251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.917198    5423 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 14:17:39.917555 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.917528    5423 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 14:17:39.917583 ip-10-0-135-142 systemd[1]: Started Kubernetes Kubelet.
Nov 01 14:17:39.918791 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.918773    5423 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 14:17:39.923698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.921543    5423 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 14:17:39.923698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.922900    5423 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 14:17:39.923698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.922917    5423 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 14:17:39.923698 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:39.922926    5423 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:17:39.923698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.922937    5423 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 14:17:39.923698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.922989    5423 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:29 +0000 UTC" deadline="2028-06-11 19:55:36.748441275 +0000 UTC"
Nov 01 14:17:39.923698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.923046    5423 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14117h37m56.825399063s"
Nov 01 14:17:39.923698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.922953    5423 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 14:17:39.924689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.924672    5423 factory.go:55] Registering systemd factory
Nov 01 14:17:39.924689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.924691    5423 factory.go:223] Registration of the systemd container factory successfully
Nov 01 14:17:39.926326 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926274    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 01 14:17:39.926425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926335    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc89564-0e45-485c-9c9f-07284715d978" volumeName="kubernetes.io/empty-dir/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-textfile" seLinuxMountContext=""
Nov 01 14:17:39.926425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926372    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52851f6-a87d-4eee-81ea-e4882afab858" volumeName="kubernetes.io/empty-dir/f52851f6-a87d-4eee-81ea-e4882afab858-tmp-dir" seLinuxMountContext=""
Nov 01 14:17:39.926425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926408    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c55c12-d679-4874-9e6f-789e7829641c" volumeName="kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926672    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a11f89e-8c0d-4412-999f-b73cd5d20e2d" volumeName="kubernetes.io/projected/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-kube-api-access-8rlx6" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926702    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="607bc669-1dea-4cec-9a9b-5d726e2ea979" volumeName="kubernetes.io/empty-dir/607bc669-1dea-4cec-9a9b-5d726e2ea979-volume-directive-shadow" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926716    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746ff76c-dd84-49d4-b178-c687185af620" volumeName="kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926729    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a370ac2-b58d-42cf-91b8-2754d966dc04" volumeName="kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926747    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52851f6-a87d-4eee-81ea-e4882afab858" volumeName="kubernetes.io/configmap/f52851f6-a87d-4eee-81ea-e4882afab858-config-volume" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926760    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c55c12-d679-4874-9e6f-789e7829641c" volumeName="kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-env-overrides" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926773    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4789a977-cfa4-479f-9e1e-bd5467d96e49" volumeName="kubernetes.io/secret/4789a977-cfa4-479f-9e1e-bd5467d96e49-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926787    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57920aa2-346d-4bc2-99a0-6e92f382fa30" volumeName="kubernetes.io/projected/57920aa2-346d-4bc2-99a0-6e92f382fa30-kube-api-access-8mfs4" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926798    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc89564-0e45-485c-9c9f-07284715d978" volumeName="kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-tls" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926815    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc007ccc-0f98-48f3-a43e-bec86a8d4523" volumeName="kubernetes.io/projected/cc007ccc-0f98-48f3-a43e-bec86a8d4523-kube-api-access-h7957" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926833    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c55c12-d679-4874-9e6f-789e7829641c" volumeName="kubernetes.io/secret/08c55c12-d679-4874-9e6f-789e7829641c-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926845    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac04470f-39e4-48f5-8d71-364e1d95f644" volumeName="kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926856    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52851f6-a87d-4eee-81ea-e4882afab858" volumeName="kubernetes.io/secret/f52851f6-a87d-4eee-81ea-e4882afab858-metrics-tls" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926884    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c55c12-d679-4874-9e6f-789e7829641c" volumeName="kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-config" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926897    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ab3520-5128-4753-81e2-463deb5298d3" volumeName="kubernetes.io/projected/34ab3520-5128-4753-81e2-463deb5298d3-kube-api-access-w8zw6" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926909    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926922    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f644ed-0d59-4762-af3b-45eefe8edc78" volumeName="kubernetes.io/projected/f9f644ed-0d59-4762-af3b-45eefe8edc78-kube-api-access-7jlct" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926748    5423 factory.go:153] Registering CRI-O factory
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926933    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f945e0d-6eb2-4c2f-ac70-e37fffe05749" volumeName="kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-daemon-config" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926945    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f945e0d-6eb2-4c2f-ac70-e37fffe05749" volumeName="kubernetes.io/projected/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-kube-api-access-qj9zk" seLinuxMountContext=""
Nov 01 14:17:39.926938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926951    5423 factory.go:223] Registration of the crio container factory successfully
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926956    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="607bc669-1dea-4cec-9a9b-5d726e2ea979" volumeName="kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926968    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926981    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-grpc-tls" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.926993    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc007ccc-0f98-48f3-a43e-bec86a8d4523" volumeName="kubernetes.io/secret/cc007ccc-0f98-48f3-a43e-bec86a8d4523-cert" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927013    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14c45109-a849-4e82-9c13-c5d1af56806e" volumeName="kubernetes.io/configmap/14c45109-a849-4e82-9c13-c5d1af56806e-serviceca" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927023    5423 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927031    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14c45109-a849-4e82-9c13-c5d1af56806e" volumeName="kubernetes.io/projected/14c45109-a849-4e82-9c13-c5d1af56806e-kube-api-access-g4wrs" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927043    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a11f89e-8c0d-4412-999f-b73cd5d20e2d" volumeName="kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-tuned" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927055    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="881f3467-ef5c-4bcc-b730-5f07644265b6" volumeName="kubernetes.io/configmap/881f3467-ef5c-4bcc-b730-5f07644265b6-iptables-alerter-script" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927066    5423 factory.go:103] Registering Raw factory
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927086    5423 manager.go:1199] Started watching for new ooms in manager
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927067    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927141    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08c55c12-d679-4874-9e6f-789e7829641c" volumeName="kubernetes.io/projected/08c55c12-d679-4874-9e6f-789e7829641c-kube-api-access-sfk4t" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927156    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="120f88bc-d7ec-43a0-8503-726fb3ca3947" volumeName="kubernetes.io/configmap/120f88bc-d7ec-43a0-8503-726fb3ca3947-konnectivity-ca" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927169    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4789a977-cfa4-479f-9e1e-bd5467d96e49" volumeName="kubernetes.io/projected/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-api-access-wnrdw" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927179    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a370ac2-b58d-42cf-91b8-2754d966dc04" volumeName="kubernetes.io/projected/9a370ac2-b58d-42cf-91b8-2754d966dc04-kube-api-access-vlswd" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927194    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc89564-0e45-485c-9c9f-07284715d978" volumeName="kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927205    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc89564-0e45-485c-9c9f-07284715d978" volumeName="kubernetes.io/projected/afc89564-0e45-485c-9c9f-07284715d978-kube-api-access-47ttc" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927217    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3585c408-fcee-40eb-9184-220f1f51a1ab" volumeName="kubernetes.io/secret/3585c408-fcee-40eb-9184-220f1f51a1ab-monitoring-plugin-cert" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927230    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57920aa2-346d-4bc2-99a0-6e92f382fa30" volumeName="kubernetes.io/empty-dir/57920aa2-346d-4bc2-99a0-6e92f382fa30-tmp-dir" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927241    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a11f89e-8c0d-4412-999f-b73cd5d20e2d" volumeName="kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-tmp" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927252    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="881f3467-ef5c-4bcc-b730-5f07644265b6" volumeName="kubernetes.io/projected/881f3467-ef5c-4bcc-b730-5f07644265b6-kube-api-access-899qc" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927263    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a370ac2-b58d-42cf-91b8-2754d966dc04" volumeName="kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927275    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/projected/a61c1192-df73-4a89-88d3-9ebb9d890827-kube-api-access-8tfh7" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927288    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f52851f6-a87d-4eee-81ea-e4882afab858" volumeName="kubernetes.io/projected/f52851f6-a87d-4eee-81ea-e4882afab858-kube-api-access-md7cd" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927302    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87bcc491-1e33-442b-b230-0cb7ae30f544" volumeName="kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927327    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/empty-dir/a61c1192-df73-4a89-88d3-9ebb9d890827-tmp" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927339    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="120f88bc-d7ec-43a0-8503-726fb3ca3947" volumeName="kubernetes.io/secret/120f88bc-d7ec-43a0-8503-726fb3ca3947-agent-certs" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927351    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3585c408-fcee-40eb-9184-220f1f51a1ab" volumeName="kubernetes.io/projected/3585c408-fcee-40eb-9184-220f1f51a1ab-kube-api-access-2vq57" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927362    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f644ed-0d59-4762-af3b-45eefe8edc78" volumeName="kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927374    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f945e0d-6eb2-4c2f-ac70-e37fffe05749" volumeName="kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cni-binary-copy" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927405    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="607bc669-1dea-4cec-9a9b-5d726e2ea979" volumeName="kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-metrics-client-ca" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927418    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746ff76c-dd84-49d4-b178-c687185af620" volumeName="kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927430    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746ff76c-dd84-49d4-b178-c687185af620" volumeName="kubernetes.io/projected/746ff76c-dd84-49d4-b178-c687185af620-kube-api-access-b5ws4" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927441    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4681a549-d1f9-4bed-82c9-21a214c885b6" volumeName="kubernetes.io/projected/4681a549-d1f9-4bed-82c9-21a214c885b6-kube-api-access-n29kd" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927453    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4789a977-cfa4-479f-9e1e-bd5467d96e49" volumeName="kubernetes.io/empty-dir/4789a977-cfa4-479f-9e1e-bd5467d96e49-data-volume" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927462    5423 manager.go:319] Starting recovery of all containers
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927475    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="607bc669-1dea-4cec-9a9b-5d726e2ea979" volumeName="kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927488    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="607bc669-1dea-4cec-9a9b-5d726e2ea979" volumeName="kubernetes.io/projected/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-api-access-8s2h6" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927499    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746ff76c-dd84-49d4-b178-c687185af620" volumeName="kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-binary-copy" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927510    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc89564-0e45-485c-9c9f-07284715d978" volumeName="kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927522    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ab3520-5128-4753-81e2-463deb5298d3" volumeName="kubernetes.io/empty-dir/34ab3520-5128-4753-81e2-463deb5298d3-tmp" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927533    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="607bc669-1dea-4cec-9a9b-5d726e2ea979" volumeName="kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-tls" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927546    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927569    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4789a977-cfa4-479f-9e1e-bd5467d96e49" volumeName="kubernetes.io/configmap/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927580    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a370ac2-b58d-42cf-91b8-2754d966dc04" volumeName="kubernetes.io/configmap/9a370ac2-b58d-42cf-91b8-2754d966dc04-metrics-client-ca" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927591    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a61c1192-df73-4a89-88d3-9ebb9d890827" volumeName="kubernetes.io/configmap/a61c1192-df73-4a89-88d3-9ebb9d890827-metrics-client-ca" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927602    5423 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc89564-0e45-485c-9c9f-07284715d978" volumeName="kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-metrics-client-ca" seLinuxMountContext=""
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927613    5423 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 14:17:39.927681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.927621    5423 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 14:17:39.936487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.936281    5423 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 14:17:39.945891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.945850    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:17:39.951750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.951725    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:17:39.952539 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.952513    5423 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 14:17:39.952539 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.952541    5423 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 14:17:39.952692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.952562    5423 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 14:17:39.952692 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:39.952606    5423 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 14:17:39.958726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.958699    5423 generic.go:356] "Generic (PLEG): container finished" podID="afc89564-0e45-485c-9c9f-07284715d978" containerID="bbd2df8f9eb3b9d1323b9fd996a8da31b040f1876950d94b43180ce13c513649" exitCode=0
Nov 01 14:17:39.965799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.965776    5423 generic.go:356] "Generic (PLEG): container finished" podID="38e27d7e3a4b8b3b59f6de6ab62a43ca" containerID="ff65d7b312a5af58db60f292c4b0e9ce8b279ee0996090f75e181102d162ffa0" exitCode=0
Nov 01 14:17:39.996085 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.996055    5423 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="09e6771ea2c45422c1a6e254ea665b74b6e14f2fc03f5af6e7852c9a40696d63" exitCode=0
Nov 01 14:17:39.996085 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.996085    5423 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="17327d36580352c5731dcc82bf7567dfd8043322c9ee6ab5542b5fefea8c2eef" exitCode=0
Nov 01 14:17:39.996246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.996094    5423 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="906521fe79472a64fe5c3a3a95fcd2c6ade9eb8d0b60a511970361c992e051b3" exitCode=0
Nov 01 14:17:39.996246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.996102    5423 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="370b212775d68ee9e8ba924c16b3a6e03b42ea4b8b913ef16b0ce0ee820b694a" exitCode=0
Nov 01 14:17:39.996246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.996107    5423 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="bae1714c666c3ccd5de43360fabdc69b38c3f87ef3541581967b4323219e537d" exitCode=0
Nov 01 14:17:39.996246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:39.996112    5423 generic.go:356] "Generic (PLEG): container finished" podID="746ff76c-dd84-49d4-b178-c687185af620" containerID="9e6faa3b2251b1c7b6daeb6f8eb03fb1ed69d785fa66b219fffac250f16cb9ea" exitCode=0
Nov 01 14:17:40.000503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.000482    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:17:40.000831 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.000805    5423 generic.go:356] "Generic (PLEG): container finished" podID="08c55c12-d679-4874-9e6f-789e7829641c" containerID="1b7c7098dae29ee5c83051f954c0bebc2ca1babb91621b5fe43d234a71764be9" exitCode=1
Nov 01 14:17:40.023016 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:40.022989    5423 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:17:40.025381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.025361    5423 manager.go:324] Recovery completed
Nov 01 14:17:40.045065 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.045007    5423 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:17:40.046934 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.046916    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:17:40.047017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.046947    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:17:40.047017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.046958    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:17:40.049459 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049441    5423 cpu_manager.go:226] "Starting" policy="none"
Nov 01 14:17:40.049459 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049454    5423 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 14:17:40.049554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049460    5423 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:17:40.049554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049473    5423 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 14:17:40.049616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049591    5423 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 14:17:40.049616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049602    5423 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 14:17:40.049675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049623    5423 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 14:17:40.049675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049629    5423 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 14:17:40.049675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049635    5423 policy_none.go:50] "Start"
Nov 01 14:17:40.049675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049643    5423 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 14:17:40.049675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049652    5423 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 14:17:40.049841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049781    5423 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 14:17:40.049841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049788    5423 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 14:17:40.049841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.049794    5423 policy_none.go:44] "Start"
Nov 01 14:17:40.050279 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.050263    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:17:40.050352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.050296    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:17:40.050352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.050313    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:17:40.052979 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:40.052963    5423 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 14:17:40.054465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.054437    5423 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 14:17:40.058171 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058157    5423 manager.go:341] "Starting Device Plugin manager"
Nov 01 14:17:40.058245 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:40.058180    5423 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 14:17:40.058245 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058190    5423 server.go:87] "Starting device plugin registration server"
Nov 01 14:17:40.058549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058534    5423 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 14:17:40.058629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058551    5423 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 14:17:40.058695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058682    5423 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 14:17:40.058808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058772    5423 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 14:17:40.058911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058818    5423 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 14:17:40.058911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058825    5423 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 14:17:40.059017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.058881    5423 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T14:17:40.058803849Z","UUID":"15f6cc6c-b627-4127-9317-be858bca8dbd","Handler":null,"Name":"","Endpoint":""}
Nov 01 14:17:40.060641 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:40.060616    5423 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 14:17:40.060808 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:40.060788    5423 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-142.us-west-2.compute.internal\" not found"
Nov 01 14:17:40.060977 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.060957    5423 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 14:17:40.060977 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.060981    5423 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 14:17:40.109671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.109646    5423 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:17:40.109811 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.109653    5423 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:17:40.109882 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.109813    5423 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 14:17:40.110870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.110852    5423 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:17:40.159124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.159096    5423 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:17:40.160093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.160072    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:17:40.160187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.160104    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:17:40.160187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.160115    5423 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-142.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:17:40.160187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.160152    5423 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.209670 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.209647    5423 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:17:40.253073 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.253020    5423 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"]
Nov 01 14:17:40.253603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.253571    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"38e27d7e3a4b8b3b59f6de6ab62a43ca","Type":"ContainerStarted","Data":"dfcea310dcec4be5e83789e17f7d04be0c9d1d8dd73f842bb4100f9fafadfe1b"}
Nov 01 14:17:40.253661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.253614    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"38e27d7e3a4b8b3b59f6de6ab62a43ca","Type":"ContainerDied","Data":"ff65d7b312a5af58db60f292c4b0e9ce8b279ee0996090f75e181102d162ffa0"}
Nov 01 14:17:40.253661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.253633    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"38e27d7e3a4b8b3b59f6de6ab62a43ca","Type":"ContainerStarted","Data":"c7d6abce1d36443e2f8e24fb0f4f96fe2ca3867067b40a422373fecd8ad0ac1a"}
Nov 01 14:17:40.253772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.253671    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"fccbcd60b09b5d4721c0b990d6d1ecb1","Type":"ContainerStarted","Data":"518a174d356f638577498f293de3cb9f37c33bff487dd7bb16a1af03d14cdafa"}
Nov 01 14:17:40.253772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.253683    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal" event={"ID":"fccbcd60b09b5d4721c0b990d6d1ecb1","Type":"ContainerStarted","Data":"4f95137e087a14eb0b1b186e49e0de878f9e03c618bfd3eafb035496c1b50fb0"}
Nov 01 14:17:40.253772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.253633    5423 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.253918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.253822    5423 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.270397 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.270373    5423 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.270485 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.270450    5423 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.316858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.316824    5423 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:17:40.316971 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:40.316869    5423 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.317218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.317205    5423 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:17:40.317280 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:40.317234    5423 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.411526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.411501    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.411663 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.411531    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.411663 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.411555    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccbcd60b09b5d4721c0b990d6d1ecb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"fccbcd60b09b5d4721c0b990d6d1ecb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.512561 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.512534    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.512712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.512571    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.512712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.512595    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccbcd60b09b5d4721c0b990d6d1ecb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"fccbcd60b09b5d4721c0b990d6d1ecb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.512712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.512650    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fccbcd60b09b5d4721c0b990d6d1ecb1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"fccbcd60b09b5d4721c0b990d6d1ecb1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.512712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.512656    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.512712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.512689    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/38e27d7e3a4b8b3b59f6de6ab62a43ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" (UID: \"38e27d7e3a4b8b3b59f6de6ab62a43ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:40.898358 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:40.898327    5423 apiserver.go:52] "Watching apiserver"
Nov 01 14:17:41.003593 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.003558    5423 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 14:17:41.004175 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.004149    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-thrgt","openshift-ingress-canary/ingress-canary-8rbwt","kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-86fr6","openshift-cluster-node-tuning-operator/tuned-4slbg","kube-system/global-pull-secret-syncer-z9bf5","openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9","openshift-insights/insights-runtime-extractor-zq7mp","openshift-image-registry/node-ca-2nh9b","openshift-monitoring/thanos-querier-bb65fcfbb-txsmv","openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5","kube-system/konnectivity-agent-g44v5","openshift-dns/node-resolver-tqbqm","openshift-monitoring/monitoring-plugin-77f4f76594-klphz","openshift-ovn-kubernetes/ovnkube-node-k8d4s","openshift-console/downloads-6dd789c8b7-sggrb","openshift-dns/dns-default-xxjqs","openshift-monitoring/node-exporter-vvzfx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal","openshift-multus/multus-wn9x6","openshift-network-operator/iptables-alerter-qfz2f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f","openshift-multus/network-metrics-daemon-9c6pv"]
Nov 01 14:17:41.012860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.012836    5423 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:41.012960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.012927    5423 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:41.025353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.025331    5423 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 14:17:41.058673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058650    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.058780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058739    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.058907 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058891    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.058982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058906    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 14:17:41.058982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058922    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.058982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058954    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.058982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058954    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.058982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058977    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 14:17:41.059191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.058977    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 14:17:41.059191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.059022    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.059191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.059086    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 14:17:41.059191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.059099    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 14:17:41.059353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.059321    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.112498 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.112472    5423 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:17:41.112628 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:41.112527    5423 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:41.113093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.113076    5423 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:17:41.113137 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:17:41.113113    5423 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-142.us-west-2.compute.internal"
Nov 01 14:17:41.115370 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115352    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-system-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.115425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115378    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cnibin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.115477 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115464    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-system-cni-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.115509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115483    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-conf\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.115509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115498    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-sys\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.115595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115517    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfs4\" (UniqueName: \"kubernetes.io/projected/57920aa2-346d-4bc2-99a0-6e92f382fa30-kube-api-access-8mfs4\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:17:41.115595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115535    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-config\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.115674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115591    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52851f6-a87d-4eee-81ea-e4882afab858-tmp-dir\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:41.115674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115650    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.115764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115674    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.115764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115717    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-kubelet\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.115764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115738    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/120f88bc-d7ec-43a0-8503-726fb3ca3947-agent-certs\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:17:41.115764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115718    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52851f6-a87d-4eee-81ea-e4882afab858-tmp-dir\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:41.115921 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115797    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-binary-copy\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.115921 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.115901    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.116022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116006    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-k8s-cni-cncf-io\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.116079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116061    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.116079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116071    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-binary-copy\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.116142 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116109    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:41.116142 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116132    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-sys\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.116211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116150    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-wtmp\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.116243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116209    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/34ab3520-5128-4753-81e2-463deb5298d3-kube-api-access-w8zw6\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:41.116273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116235    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-log-socket\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.116273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116260    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-bin\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.116325 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116276    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08c55c12-d679-4874-9e6f-789e7829641c-ovn-node-metrics-cert\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.116448 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116403    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ttc\" (UniqueName: \"kubernetes.io/projected/afc89564-0e45-485c-9c9f-07284715d978-kube-api-access-47ttc\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.116448 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116441    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cni-binary-copy\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.116630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116461    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/120f88bc-d7ec-43a0-8503-726fb3ca3947-konnectivity-ca\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:17:41.116630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116490    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.116630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116514    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:41.116630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116538    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899qc\" (UniqueName: \"kubernetes.io/projected/881f3467-ef5c-4bcc-b730-5f07644265b6-kube-api-access-899qc\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:17:41.116630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116564    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.116817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116632    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/881f3467-ef5c-4bcc-b730-5f07644265b6-iptables-alerter-script\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:17:41.116817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116679    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cni-binary-copy\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.116817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116697    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.116817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116754    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3585c408-fcee-40eb-9184-220f1f51a1ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:41.116817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116762    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.116817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116782    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-netns\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.116817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116808    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61c1192-df73-4a89-88d3-9ebb9d890827-tmp\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.117040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116841    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.117040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116861    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-bin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.117040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116931    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61c1192-df73-4a89-88d3-9ebb9d890827-tmp\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.117040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116892    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.117040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116973    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-device-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.117040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.116989    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-run\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.117040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117004    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-etc-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.117040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117019    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/881f3467-ef5c-4bcc-b730-5f07644265b6-host-slash\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:17:41.117323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117059    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57920aa2-346d-4bc2-99a0-6e92f382fa30-hosts-file\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:17:41.117323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117078    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f52851f6-a87d-4eee-81ea-e4882afab858-config-volume\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:41.117323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117124    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jlct\" (UniqueName: \"kubernetes.io/projected/f9f644ed-0d59-4762-af3b-45eefe8edc78-kube-api-access-7jlct\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:41.117323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117192    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-netd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.117323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117309    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4789a977-cfa4-479f-9e1e-bd5467d96e49-crio-socket\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.117549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117340    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.117549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117371    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-daemon-config\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.117549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117432    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.117549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117499    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29kd\" (UniqueName: \"kubernetes.io/projected/4681a549-d1f9-4bed-82c9-21a214c885b6-kube-api-access-n29kd\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.117549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117528    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrdw\" (UniqueName: \"kubernetes.io/projected/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-api-access-wnrdw\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.117775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117562    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wrs\" (UniqueName: \"kubernetes.io/projected/14c45109-a849-4e82-9c13-c5d1af56806e-kube-api-access-g4wrs\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:17:41.117775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117589    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ab3520-5128-4753-81e2-463deb5298d3-tmp\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:41.117775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117649    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-daemon-config\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.117775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117662    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34ab3520-5128-4753-81e2-463deb5298d3-tmp\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:41.117775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117728    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14c45109-a849-4e82-9c13-c5d1af56806e-serviceca\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117765    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61c1192-df73-4a89-88d3-9ebb9d890827-metrics-client-ca\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117817    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysconfig\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117841    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-netns\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117914    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-node-log\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117942    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.117968    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-tmp\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118008    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57920aa2-346d-4bc2-99a0-6e92f382fa30-tmp-dir\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118040    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2h6\" (UniqueName: \"kubernetes.io/projected/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-api-access-8s2h6\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118046    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-tmp\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118056    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118085    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc007ccc-0f98-48f3-a43e-bec86a8d4523-cert\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118110    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-textfile\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118142    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vq57\" (UniqueName: \"kubernetes.io/projected/3585c408-fcee-40eb-9184-220f1f51a1ab-kube-api-access-2vq57\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118117    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57920aa2-346d-4bc2-99a0-6e92f382fa30-tmp-dir\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118175    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-multus\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118210    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rlx6\" (UniqueName: \"kubernetes.io/projected/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-kube-api-access-8rlx6\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118218    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-textfile\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118245    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118272    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118297    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-systemd\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118302    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14c45109-a849-4e82-9c13-c5d1af56806e-serviceca\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118320    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-slash\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118413    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118445    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118469    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118486    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfk4t\" (UniqueName: \"kubernetes.io/projected/08c55c12-d679-4874-9e6f-789e7829641c-kube-api-access-sfk4t\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118502    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/607bc669-1dea-4cec-9a9b-5d726e2ea979-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118520    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-grpc-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118549    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118577    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-kubernetes\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118602    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-kubelet\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118604    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/607bc669-1dea-4cec-9a9b-5d726e2ea979-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118664    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-systemd-units\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118696    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4789a977-cfa4-479f-9e1e-bd5467d96e49-data-volume\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118700    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/746ff76c-dd84-49d4-b178-c687185af620-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118740    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-root\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118795    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4789a977-cfa4-479f-9e1e-bd5467d96e49-data-volume\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118819    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118838    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-modprobe-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118852    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-host\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118868    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-ovn\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118893    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-env-overrides\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118912    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-socket-dir-parent\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118937    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-cnibin\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.118962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.118959    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-script-lib\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119029    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4789a977-cfa4-479f-9e1e-bd5467d96e49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119101    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119170    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-sys-fs\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119203    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-var-lib-kubelet\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119251    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a370ac2-b58d-42cf-91b8-2754d966dc04-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119289    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ws4\" (UniqueName: \"kubernetes.io/projected/746ff76c-dd84-49d4-b178-c687185af620-kube-api-access-b5ws4\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119314    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-tuned\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119350    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-systemd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119376    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119423    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-conf-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119472    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-tuned\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119456    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119555    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7957\" (UniqueName: \"kubernetes.io/projected/cc007ccc-0f98-48f3-a43e-bec86a8d4523-kube-api-access-h7957\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119587    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-socket-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119615    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7cd\" (UniqueName: \"kubernetes.io/projected/f52851f6-a87d-4eee-81ea-e4882afab858-kube-api-access-md7cd\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119646    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-metrics-client-ca\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119691    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-hostroot\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.119808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119800    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-etc-kubernetes\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119819    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-var-lib-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119838    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-registration-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119859    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-multus-certs\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119900    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfh7\" (UniqueName: \"kubernetes.io/projected/a61c1192-df73-4a89-88d3-9ebb9d890827-kube-api-access-8tfh7\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119933    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-kubelet-config\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119953    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlswd\" (UniqueName: \"kubernetes.io/projected/9a370ac2-b58d-42cf-91b8-2754d966dc04-kube-api-access-vlswd\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.119972    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.120007    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.120047    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f52851f6-a87d-4eee-81ea-e4882afab858-metrics-tls\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.120073    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-tls\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.120100    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-lib-modules\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.120131    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-dbus\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.120155    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c45109-a849-4e82-9c13-c5d1af56806e-host\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:17:41.120406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.120217    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9zk\" (UniqueName: \"kubernetes.io/projected/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-kube-api-access-qj9zk\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.140377 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.140354    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 14:17:41.140623 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.140601    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 14:17:41.140722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.140676    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 14:17:41.140722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.140686    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 14:17:41.140722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.140709    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 14:17:41.140838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.140812    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-9qhws\""
Nov 01 14:17:41.141156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141141    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7pw86\""
Nov 01 14:17:41.141192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141170    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 14:17:41.141298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141284    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 14:17:41.141358 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141307    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h4vvk\""
Nov 01 14:17:41.141426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141420    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 14:17:41.141547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141529    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mpq57\""
Nov 01 14:17:41.141595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141571    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lp9km\""
Nov 01 14:17:41.141689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141669    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 14:17:41.141689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141679    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 14:17:41.141689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141688    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-smpqh\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141696    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5wqpw\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141671    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj7jb\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141706    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141714    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjzzz\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141729    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141765    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mxbff\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141782    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141790    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141801    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2nj9\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141824    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141837    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141851    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141853    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c07asod0mkf0b\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141870    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fnkb\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141881    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8xv6q\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141882    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 14:17:41.141918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141925    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.141962    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9p5x8\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.142009    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ftvdm\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.142056    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-24rb9\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.142111    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x2zjc\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.142140    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xrrrs\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.142180    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m46fn\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.142232    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k29xh\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.142254    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 14:17:41.142349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.142276    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v4966\""
Nov 01 14:17:41.146665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.146644    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:41.146665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.146658    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.146765 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.146696    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/120f88bc-d7ec-43a0-8503-726fb3ca3947-agent-certs\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:17:41.146765 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.146745    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08c55c12-d679-4874-9e6f-789e7829641c-ovn-node-metrics-cert\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.146835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.146766    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.146905 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.146889    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87bcc491-1e33-442b-b230-0cb7ae30f544-original-pull-secret\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:41.147038 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.147026    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3585c408-fcee-40eb-9184-220f1f51a1ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:41.147136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.147124    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.147841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.147827    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.149047 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.149001    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.149161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.149085    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f644ed-0d59-4762-af3b-45eefe8edc78-metrics-certs\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:41.149161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.149114    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.149161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.149155    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc007ccc-0f98-48f3-a43e-bec86a8d4523-cert\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:41.149321 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.149175    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-grpc-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.149321 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.149281    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4789a977-cfa4-479f-9e1e-bd5467d96e49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.149505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.149489    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-tls\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.150275 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.150251    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a61c1192-df73-4a89-88d3-9ebb9d890827-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.150342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.150297    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f52851f6-a87d-4eee-81ea-e4882afab858-metrics-tls\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:41.150342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.150319    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a370ac2-b58d-42cf-91b8-2754d966dc04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:41.150424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.150412    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-tls\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.158650 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.158623    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.158650 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.158648    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.158799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.158626    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 14:17:41.159726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.158826    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 14:17:41.159726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.159139    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.159726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.159288    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.159726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.159366    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-env-overrides\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.159940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.159920    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 14:17:41.160086 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160070    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 14:17:41.160155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160142    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.160505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160257    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 14:17:41.160505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160306    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.160505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160366    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.160505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160443    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 14:17:41.160743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160554    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.160743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160705    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.160743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160737    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.160874 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160749    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 14:17:41.160874 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160786    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.160960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.160933    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.161074 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.161061    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.161130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.161106    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.161175 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.161129    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 14:17:41.161218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.161199    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 14:17:41.161475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.161427    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 14:17:41.161475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.161458    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 14:17:41.161592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.161504    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 14:17:41.166069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.166052    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-config\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.166069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.166059    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.166669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.166651    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.166749 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.166735    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/120f88bc-d7ec-43a0-8503-726fb3ca3947-konnectivity-ca\") pod \"konnectivity-agent-g44v5\" (UID: \"120f88bc-d7ec-43a0-8503-726fb3ca3947\") " pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:17:41.166908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.166894    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/881f3467-ef5c-4bcc-b730-5f07644265b6-iptables-alerter-script\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:17:41.166990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.166940    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.167785 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.167768    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/607bc669-1dea-4cec-9a9b-5d726e2ea979-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.167898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.167865    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f52851f6-a87d-4eee-81ea-e4882afab858-config-volume\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:41.169065 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.169047    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a61c1192-df73-4a89-88d3-9ebb9d890827-metrics-client-ca\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.169253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.169240    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08c55c12-d679-4874-9e6f-789e7829641c-ovnkube-script-lib\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.170229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.170206    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a370ac2-b58d-42cf-91b8-2754d966dc04-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:41.170283 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.170247    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc89564-0e45-485c-9c9f-07284715d978-metrics-client-ca\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.171923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.171906    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfs4\" (UniqueName: \"kubernetes.io/projected/57920aa2-346d-4bc2-99a0-6e92f382fa30-kube-api-access-8mfs4\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:17:41.172215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.172199    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/34ab3520-5128-4753-81e2-463deb5298d3-kube-api-access-w8zw6\") pod \"downloads-6dd789c8b7-sggrb\" (UID: \"34ab3520-5128-4753-81e2-463deb5298d3\") " pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:17:41.172291 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.172271    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ttc\" (UniqueName: \"kubernetes.io/projected/afc89564-0e45-485c-9c9f-07284715d978-kube-api-access-47ttc\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.172354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.172339    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-899qc\" (UniqueName: \"kubernetes.io/projected/881f3467-ef5c-4bcc-b730-5f07644265b6-kube-api-access-899qc\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:17:41.172994 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.172976    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrdw\" (UniqueName: \"kubernetes.io/projected/4789a977-cfa4-479f-9e1e-bd5467d96e49-kube-api-access-wnrdw\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.173072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.173021    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29kd\" (UniqueName: \"kubernetes.io/projected/4681a549-d1f9-4bed-82c9-21a214c885b6-kube-api-access-n29kd\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.173072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.173045    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wrs\" (UniqueName: \"kubernetes.io/projected/14c45109-a849-4e82-9c13-c5d1af56806e-kube-api-access-g4wrs\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:17:41.173161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.173153    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jlct\" (UniqueName: \"kubernetes.io/projected/f9f644ed-0d59-4762-af3b-45eefe8edc78-kube-api-access-7jlct\") pod \"network-metrics-daemon-9c6pv\" (UID: \"f9f644ed-0d59-4762-af3b-45eefe8edc78\") " pod="openshift-multus/network-metrics-daemon-9c6pv"
Nov 01 14:17:41.177557 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.177529    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfk4t\" (UniqueName: \"kubernetes.io/projected/08c55c12-d679-4874-9e6f-789e7829641c-kube-api-access-sfk4t\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.177557 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.177545    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhn8\" (UniqueName: \"kubernetes.io/projected/ac04470f-39e4-48f5-8d71-364e1d95f644-kube-api-access-xxhn8\") pod \"network-check-target-thrgt\" (UID: \"ac04470f-39e4-48f5-8d71-364e1d95f644\") " pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:41.177654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.177532    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rlx6\" (UniqueName: \"kubernetes.io/projected/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-kube-api-access-8rlx6\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.177654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.177607    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7cd\" (UniqueName: \"kubernetes.io/projected/f52851f6-a87d-4eee-81ea-e4882afab858-kube-api-access-md7cd\") pod \"dns-default-xxjqs\" (UID: \"f52851f6-a87d-4eee-81ea-e4882afab858\") " pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:41.177723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.177693    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2h6\" (UniqueName: \"kubernetes.io/projected/607bc669-1dea-4cec-9a9b-5d726e2ea979-kube-api-access-8s2h6\") pod \"kube-state-metrics-5dc8d7d89b-jfqz9\" (UID: \"607bc669-1dea-4cec-9a9b-5d726e2ea979\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jfqz9"
Nov 01 14:17:41.177786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.177771    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vq57\" (UniqueName: \"kubernetes.io/projected/3585c408-fcee-40eb-9184-220f1f51a1ab-kube-api-access-2vq57\") pod \"monitoring-plugin-77f4f76594-klphz\" (UID: \"3585c408-fcee-40eb-9184-220f1f51a1ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:41.190792 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.190774    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ws4\" (UniqueName: \"kubernetes.io/projected/746ff76c-dd84-49d4-b178-c687185af620-kube-api-access-b5ws4\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.190871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.190836    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9zk\" (UniqueName: \"kubernetes.io/projected/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-kube-api-access-qj9zk\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.191097 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.191081    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tfh7\" (UniqueName: \"kubernetes.io/projected/a61c1192-df73-4a89-88d3-9ebb9d890827-kube-api-access-8tfh7\") pod \"thanos-querier-bb65fcfbb-txsmv\" (UID: \"a61c1192-df73-4a89-88d3-9ebb9d890827\") " pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:41.191172 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.191150    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7957\" (UniqueName: \"kubernetes.io/projected/cc007ccc-0f98-48f3-a43e-bec86a8d4523-kube-api-access-h7957\") pod \"ingress-canary-8rbwt\" (UID: \"cc007ccc-0f98-48f3-a43e-bec86a8d4523\") " pod="openshift-ingress-canary/ingress-canary-8rbwt"
Nov 01 14:17:41.191214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.191150    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlswd\" (UniqueName: \"kubernetes.io/projected/9a370ac2-b58d-42cf-91b8-2754d966dc04-kube-api-access-vlswd\") pod \"openshift-state-metrics-7bd7489957-k79t5\" (UID: \"9a370ac2-b58d-42cf-91b8-2754d966dc04\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-k79t5"
Nov 01 14:17:41.221526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221497    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-modprobe-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.221526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221530    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-host\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221546    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-ovn\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221569    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-socket-dir-parent\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221591    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-cnibin\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221627    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-sys-fs\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221635    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-ovn\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221645    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-socket-dir-parent\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221635    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-host\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221669    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-cnibin\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221645    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-var-lib-kubelet\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221697    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-var-lib-kubelet\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.221717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221711    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-systemd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221728    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221764    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-conf-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221772    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-systemd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221786    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221682    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-modprobe-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221711    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-sys-fs\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221811    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-multus-conf-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221788    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221833    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221843    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-socket-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221859    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-hostroot\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221873    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-etc-kubernetes\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221887    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-var-lib-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221915    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-socket-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221938    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-registration-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221951    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-var-lib-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221959    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-multus-certs\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221945    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-hostroot\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221956    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-etc-kubernetes\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221983    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-registration-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221990    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-multus-certs\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.221976    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-kubelet-config\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222029    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-lib-modules\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222046    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-kubelet-config\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:41.222054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222060    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-dbus\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222080    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c45109-a849-4e82-9c13-c5d1af56806e-host\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222102    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-system-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222119    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-lib-modules\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222115    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cnibin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222144    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87bcc491-1e33-442b-b230-0cb7ae30f544-dbus\") pod \"global-pull-secret-syncer-z9bf5\" (UID: \"87bcc491-1e33-442b-b230-0cb7ae30f544\") " pod="kube-system/global-pull-secret-syncer-z9bf5"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222155    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-system-cni-dir\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222163    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14c45109-a849-4e82-9c13-c5d1af56806e-host\") pod \"node-ca-2nh9b\" (UID: \"14c45109-a849-4e82-9c13-c5d1af56806e\") " pod="openshift-image-registry/node-ca-2nh9b"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222173    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-cnibin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222190    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-system-cni-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222211    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-conf\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222225    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-sys\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222234    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/746ff76c-dd84-49d4-b178-c687185af620-system-cni-dir\") pod \"multus-additional-cni-plugins-86fr6\" (UID: \"746ff76c-dd84-49d4-b178-c687185af620\") " pod="openshift-multus/multus-additional-cni-plugins-86fr6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222243    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-kubelet\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222263    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-kubelet\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222271    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222293    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-sys\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222295    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-k8s-cni-cncf-io\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222314    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-run-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222319    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-sys\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222334    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-k8s-cni-cncf-io\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222337    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-conf\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222341    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-wtmp\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222356    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-sys\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222371    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-log-socket\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222381    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-node-exporter-wtmp\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222438    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-bin\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222449    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-log-socket\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222463    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-netns\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222474    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-bin\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222478    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222493    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-bin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222497    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-run-netns\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222507    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-device-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222519    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222521    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-run\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222540    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-etc-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222542    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-bin\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222552    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-run\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222575    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-etc-openvswitch\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222578    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-device-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222596    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/881f3467-ef5c-4bcc-b730-5f07644265b6-host-slash\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222615    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57920aa2-346d-4bc2-99a0-6e92f382fa30-hosts-file\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222641    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-netd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222654    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/881f3467-ef5c-4bcc-b730-5f07644265b6-host-slash\") pod \"iptables-alerter-qfz2f\" (UID: \"881f3467-ef5c-4bcc-b730-5f07644265b6\") " pod="openshift-network-operator/iptables-alerter-qfz2f"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222663    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57920aa2-346d-4bc2-99a0-6e92f382fa30-hosts-file\") pod \"node-resolver-tqbqm\" (UID: \"57920aa2-346d-4bc2-99a0-6e92f382fa30\") " pod="openshift-dns/node-resolver-tqbqm"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222668    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4789a977-cfa4-479f-9e1e-bd5467d96e49-crio-socket\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222687    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-cni-netd\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222693    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysconfig\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222729    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-netns\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222776    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4789a977-cfa4-479f-9e1e-bd5467d96e49-crio-socket\") pod \"insights-runtime-extractor-zq7mp\" (UID: \"4789a977-cfa4-479f-9e1e-bd5467d96e49\") " pod="openshift-insights/insights-runtime-extractor-zq7mp"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222816    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-node-log\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222826    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysconfig\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222832    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-run-netns\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222836    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-node-log\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222837    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.222834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222863    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222874    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-multus\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222893    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-systemd\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222911    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f945e0d-6eb2-4c2f-ac70-e37fffe05749-host-var-lib-cni-multus\") pod \"multus-wn9x6\" (UID: \"2f945e0d-6eb2-4c2f-ac70-e37fffe05749\") " pod="openshift-multus/multus-wn9x6"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222914    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-slash\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222937    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222942    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-systemd\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222961    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222977    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-slash\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.222983    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223009    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-kubernetes\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223012    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223024    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-kubelet\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223038    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-systemd-units\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223047    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4681a549-d1f9-4bed-82c9-21a214c885b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-b2g6f\" (UID: \"4681a549-d1f9-4bed-82c9-21a214c885b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b2g6f"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223068    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-root\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223096    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-systemd-units\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223062    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-sysctl-d\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223096    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08c55c12-d679-4874-9e6f-789e7829641c-host-kubelet\") pod \"ovnkube-node-k8d4s\" (UID: \"08c55c12-d679-4874-9e6f-789e7829641c\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223070    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a11f89e-8c0d-4412-999f-b73cd5d20e2d-etc-kubernetes\") pod \"tuned-4slbg\" (UID: \"1a11f89e-8c0d-4412-999f-b73cd5d20e2d\") " pod="openshift-cluster-node-tuning-operator/tuned-4slbg"
Nov 01 14:17:41.224564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.223146    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc89564-0e45-485c-9c9f-07284715d978-root\") pod \"node-exporter-vvzfx\" (UID: \"afc89564-0e45-485c-9c9f-07284715d978\") " pod="openshift-monitoring/node-exporter-vvzfx"
Nov 01 14:17:41.318687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.318669    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:17:41.588631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.588598    5423 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44v5_120f88bc-d7ec-43a0-8503-726fb3ca3947/konnectivity-agent/0.log" limit=1024
Nov 01 14:17:41.806979 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:41.806947    5423 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g44v5_120f88bc-d7ec-43a0-8503-726fb3ca3947/konnectivity-agent/0.log" limit=1024
Nov 01 14:17:42.331496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:42.331466    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xxjqs"
Nov 01 14:17:42.444440 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:42.444411    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:42.450871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:42.450844    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bb65fcfbb-txsmv"
Nov 01 14:17:43.608776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:43.608750    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:45.489871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.489839    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-65bc7f65df-tb7mn"]
Nov 01 14:17:45.493053 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.493025    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.552634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.552608    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746cfe66-08c0-4eee-98c8-0eddb4d79df2-tmp\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.552769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.552642    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qvv\" (UniqueName: \"kubernetes.io/projected/746cfe66-08c0-4eee-98c8-0eddb4d79df2-kube-api-access-68qvv\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.552769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.552667    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-oauth-config\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.552769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.552688    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-config\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.552897 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.552842    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-serving-cert\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.552897 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.552881    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-service-ca\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.552961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.552909    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-oauth-serving-cert\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.573507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.573476    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:45.578536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.578508    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-klphz"
Nov 01 14:17:45.596571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.596498    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jl52p\""
Nov 01 14:17:45.596571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.596509    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 14:17:45.596571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.596513    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 14:17:45.596571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.596571    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 14:17:45.596839 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.596602    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 14:17:45.596839 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.596608    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 14:17:45.597175 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.597149    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65bc7f65df-tb7mn"]
Nov 01 14:17:45.653454 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.653427    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-serving-cert\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.653454 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.653458    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-service-ca\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.653685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.653480    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-oauth-serving-cert\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.653685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.653538    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746cfe66-08c0-4eee-98c8-0eddb4d79df2-tmp\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.653882 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.653858    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qvv\" (UniqueName: \"kubernetes.io/projected/746cfe66-08c0-4eee-98c8-0eddb4d79df2-kube-api-access-68qvv\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.653943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.653929    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-oauth-config\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.653992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.653962    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-config\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.653992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.653980    5423 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 14:17:45.654096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.654014    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746cfe66-08c0-4eee-98c8-0eddb4d79df2-tmp\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.654262 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.654238    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-service-ca\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.654376 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.654271    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-oauth-serving-cert\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.654556 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.654537    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-config\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.656660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.656632    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-serving-cert\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.657121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.657096    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-oauth-config\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.715706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.715678    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qvv\" (UniqueName: \"kubernetes.io/projected/746cfe66-08c0-4eee-98c8-0eddb4d79df2-kube-api-access-68qvv\") pod \"console-65bc7f65df-tb7mn\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") " pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:45.806348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:45.806322    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:46.232379 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:46.232343    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746cfe66_08c0_4eee_98c8_0eddb4d79df2.slice/crio-c04327d404d391d968dd789ade52288689695be972bc6a3806143a8ad2badddf WatchSource:0}: Error finding container c04327d404d391d968dd789ade52288689695be972bc6a3806143a8ad2badddf: Status 404 returned error can't find the container with id c04327d404d391d968dd789ade52288689695be972bc6a3806143a8ad2badddf
Nov 01 14:17:46.232669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:46.232632    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65bc7f65df-tb7mn"]
Nov 01 14:17:47.038988 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:47.038952    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bc7f65df-tb7mn" event={"ID":"746cfe66-08c0-4eee-98c8-0eddb4d79df2","Type":"ContainerStarted","Data":"c04327d404d391d968dd789ade52288689695be972bc6a3806143a8ad2badddf"}
Nov 01 14:17:48.316760 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:48.316727    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:48.317216 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:48.316999    5423 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 14:17:50.005691 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:50.005657    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:17:50.052440 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:50.052378    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bc7f65df-tb7mn" event={"ID":"746cfe66-08c0-4eee-98c8-0eddb4d79df2","Type":"ContainerStarted","Data":"9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725"}
Nov 01 14:17:50.231857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:50.231742    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-65bc7f65df-tb7mn" podStartSLOduration=1.597795992 podStartE2EDuration="5.231726363s" totalImagesPullingTime="3.633930371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:50.231549978 +0000 UTC m=+10.397035276" watchObservedRunningTime="2026-11-01 14:17:50.231726363 +0000 UTC m=+10.397211663"
Nov 01 14:17:51.406437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:51.406409    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g44v5"
Nov 01 14:17:53.485581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:53.485542    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:56.706458 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:56.706429    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8d4s"
Nov 01 14:17:57.221079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.221052    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c765b696f-bbnvm"]
Nov 01 14:17:57.228180 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.228145    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.345760 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.345721    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-serving-cert\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.345911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.345774    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-oauth-serving-cert\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.345911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.345805    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-oauth-config\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.345911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.345834    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8kv\" (UniqueName: \"kubernetes.io/projected/9b96d6ee-793c-4c5e-b02a-70841aca81ad-kube-api-access-sc8kv\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.345911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.345872    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b96d6ee-793c-4c5e-b02a-70841aca81ad-tmp\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.345911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.345908    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-trusted-ca-bundle\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.346113 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.345941    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-config\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.346113 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.345971    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-service-ca\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.434606 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.434573    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 14:17:57.435088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.435063    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c765b696f-bbnvm"]
Nov 01 14:17:57.446737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.446713    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-trusted-ca-bundle\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.446837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.446757    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-config\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.446837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.446779    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-service-ca\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.446837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.446823    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-serving-cert\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.447004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.446843    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-oauth-serving-cert\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.447004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.446861    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-oauth-config\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.447004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.446879    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8kv\" (UniqueName: \"kubernetes.io/projected/9b96d6ee-793c-4c5e-b02a-70841aca81ad-kube-api-access-sc8kv\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.447004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.446903    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b96d6ee-793c-4c5e-b02a-70841aca81ad-tmp\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.447250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.447233    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b96d6ee-793c-4c5e-b02a-70841aca81ad-tmp\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.447551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.447529    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-config\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.447738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.447717    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-trusted-ca-bundle\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.447834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.447815    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-service-ca\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.448082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.448064    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-oauth-serving-cert\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.450282 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.450265    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-oauth-config\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.450517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.450495    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-serving-cert\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.508781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.508707    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8kv\" (UniqueName: \"kubernetes.io/projected/9b96d6ee-793c-4c5e-b02a-70841aca81ad-kube-api-access-sc8kv\") pod \"console-6c765b696f-bbnvm\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") " pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.540540 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.540312    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:17:57.914688 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.914660    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c765b696f-bbnvm"]
Nov 01 14:17:57.915270 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:17:57.915235    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b96d6ee_793c_4c5e_b02a_70841aca81ad.slice/crio-70b15ec53e663c52a2f239dd78eca6ee7002f1d92744f9bb14f0148fd0c81388 WatchSource:0}: Error finding container 70b15ec53e663c52a2f239dd78eca6ee7002f1d92744f9bb14f0148fd0c81388: Status 404 returned error can't find the container with id 70b15ec53e663c52a2f239dd78eca6ee7002f1d92744f9bb14f0148fd0c81388
Nov 01 14:17:57.936500 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.935924    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:57.941202 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:57.941178    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:58.086670 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:58.086636    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c765b696f-bbnvm" event={"ID":"9b96d6ee-793c-4c5e-b02a-70841aca81ad","Type":"ContainerStarted","Data":"b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d"}
Nov 01 14:17:58.086858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:58.086687    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c765b696f-bbnvm" event={"ID":"9b96d6ee-793c-4c5e-b02a-70841aca81ad","Type":"ContainerStarted","Data":"70b15ec53e663c52a2f239dd78eca6ee7002f1d92744f9bb14f0148fd0c81388"}
Nov 01 14:17:58.090675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:58.090652    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:17:58.203404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:17:58.203288    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c765b696f-bbnvm" podStartSLOduration=1.203272054 podStartE2EDuration="1.203272054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:58.20308259 +0000 UTC m=+18.368567888" watchObservedRunningTime="2026-11-01 14:17:58.203272054 +0000 UTC m=+18.368757352"
Nov 01 14:18:00.105691 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:00.105587    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sggrb" event={"ID":"34ab3520-5128-4753-81e2-463deb5298d3","Type":"ContainerStarted","Data":"8b7a5fdabf8b221739b9efd58f2caf935619c265b480f611e1da3cf3d84e8cd1"}
Nov 01 14:18:00.106214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:00.106156    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:18:00.127807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:00.127783    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-sggrb"
Nov 01 14:18:05.152864 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:18:05.152826    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:18:06.602951 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:18:06.602912    5423 webhook.go:269] Failed to make webhook authorizer request: Post "https://a2ba49ef185a5444dbb27d29e6baf327-c3ad9244eeee9bec.elb.us-east-2.amazonaws.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews": context canceled
Nov 01 14:18:06.602951 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:18:06.602950    5423 server.go:358] "Authorization error" err="Post \"https://a2ba49ef185a5444dbb27d29e6baf327-c3ad9244eeee9bec.elb.us-east-2.amazonaws.com:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": context canceled" user="system:serviceaccount:openshift-monitoring:prometheus-k8s" verb="get" resource="nodes" subresource="metrics"
Nov 01 14:18:07.541327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:07.541289    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:18:07.541522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:07.541408    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:18:07.546822 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:07.546795    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:18:08.143904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:08.143875    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:18:08.310307 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:08.310273    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65bc7f65df-tb7mn"]
Nov 01 14:18:08.900533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:08.900501    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58ccd5b85b-svg4v"]
Nov 01 14:18:08.961935 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:08.961904    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58ccd5b85b-svg4v"]
Nov 01 14:18:08.962082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:08.962018    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.048638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.048606    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-serving-cert\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.048638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.048636    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-config\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.048861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.048658    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-oauth-config\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.048861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.048683    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-service-ca\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.048861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.048843    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-trusted-ca-bundle\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.048972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.048863    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng55b\" (UniqueName: \"kubernetes.io/projected/369e3825-b4ec-4a0c-afaf-a58987feabdb-kube-api-access-ng55b\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.048972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.048882    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-oauth-serving-cert\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.048972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.048943    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369e3825-b4ec-4a0c-afaf-a58987feabdb-tmp\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150021 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.149996    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-serving-cert\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150021 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.150023    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-config\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.150045    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-oauth-config\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.150072    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-service-ca\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.150125    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-trusted-ca-bundle\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.150148    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng55b\" (UniqueName: \"kubernetes.io/projected/369e3825-b4ec-4a0c-afaf-a58987feabdb-kube-api-access-ng55b\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.150178    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-oauth-serving-cert\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.150204    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369e3825-b4ec-4a0c-afaf-a58987feabdb-tmp\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.150845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.150667    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369e3825-b4ec-4a0c-afaf-a58987feabdb-tmp\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.151201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.151176    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-config\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.151201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.151197    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-service-ca\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.151326 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.151176    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-trusted-ca-bundle\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.151326 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.151245    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-oauth-serving-cert\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.152674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.152656    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-oauth-config\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.152765 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.152721    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-serving-cert\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.206031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.206009    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng55b\" (UniqueName: \"kubernetes.io/projected/369e3825-b4ec-4a0c-afaf-a58987feabdb-kube-api-access-ng55b\") pod \"console-58ccd5b85b-svg4v\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") " pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.289400 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.289362    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:09.682796 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:18:09.682749    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369e3825_b4ec_4a0c_afaf_a58987feabdb.slice/crio-3d5dc5abb42aaad0fb15c27b720b5ea71cf344a7a722eb9fa5bcc5ee7ba98d53 WatchSource:0}: Error finding container 3d5dc5abb42aaad0fb15c27b720b5ea71cf344a7a722eb9fa5bcc5ee7ba98d53: Status 404 returned error can't find the container with id 3d5dc5abb42aaad0fb15c27b720b5ea71cf344a7a722eb9fa5bcc5ee7ba98d53
Nov 01 14:18:09.683433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:09.683301    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58ccd5b85b-svg4v"]
Nov 01 14:18:10.008953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:10.008881    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-thrgt"
Nov 01 14:18:10.148475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:10.148440    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ccd5b85b-svg4v" event={"ID":"369e3825-b4ec-4a0c-afaf-a58987feabdb","Type":"ContainerStarted","Data":"11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8"}
Nov 01 14:18:10.148660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:10.148496    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ccd5b85b-svg4v" event={"ID":"369e3825-b4ec-4a0c-afaf-a58987feabdb","Type":"ContainerStarted","Data":"3d5dc5abb42aaad0fb15c27b720b5ea71cf344a7a722eb9fa5bcc5ee7ba98d53"}
Nov 01 14:18:10.263097 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:10.262985    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58ccd5b85b-svg4v" podStartSLOduration=2.262971551 podStartE2EDuration="2.262971551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:18:10.262714471 +0000 UTC m=+30.428199768" watchObservedRunningTime="2026-11-01 14:18:10.262971551 +0000 UTC m=+30.428456849"
Nov 01 14:18:19.290414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:19.290370    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:19.290974 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:19.290433    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:19.296131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:19.296108    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:20.193687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:20.193658    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:18:20.360455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:20.360409    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c765b696f-bbnvm"]
Nov 01 14:18:33.329514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.329458    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-65bc7f65df-tb7mn" podUID="746cfe66-08c0-4eee-98c8-0eddb4d79df2" containerName="console" containerID="cri-o://9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725" gracePeriod=15
Nov 01 14:18:33.606329 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.606245    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:18:33.747163 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747130    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746cfe66-08c0-4eee-98c8-0eddb4d79df2-kube-api-access-68qvv\" (UniqueName: \"kubernetes.io/projected/746cfe66-08c0-4eee-98c8-0eddb4d79df2-kube-api-access-68qvv\") pod \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") "
Nov 01 14:18:33.747344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747173    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-service-ca\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-service-ca\") pod \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") "
Nov 01 14:18:33.747344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747207    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-config\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-config\") pod \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") "
Nov 01 14:18:33.747344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747234    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-serving-cert\") pod \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") "
Nov 01 14:18:33.747344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747289    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-oauth-serving-cert\") pod \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") "
Nov 01 14:18:33.747564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747365    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746cfe66-08c0-4eee-98c8-0eddb4d79df2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/746cfe66-08c0-4eee-98c8-0eddb4d79df2-tmp\") pod \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") "
Nov 01 14:18:33.747564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747437    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-oauth-config\") pod \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\" (UID: \"746cfe66-08c0-4eee-98c8-0eddb4d79df2\") "
Nov 01 14:18:33.747716 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747687    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-config" pod "746cfe66-08c0-4eee-98c8-0eddb4d79df2" (UID: "746cfe66-08c0-4eee-98c8-0eddb4d79df2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:33.747784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747689    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746cfe66-08c0-4eee-98c8-0eddb4d79df2-tmp" pod "746cfe66-08c0-4eee-98c8-0eddb4d79df2" (UID: "746cfe66-08c0-4eee-98c8-0eddb4d79df2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:18:33.747784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747700    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-oauth-serving-cert" pod "746cfe66-08c0-4eee-98c8-0eddb4d79df2" (UID: "746cfe66-08c0-4eee-98c8-0eddb4d79df2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:33.747891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.747780    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-service-ca" pod "746cfe66-08c0-4eee-98c8-0eddb4d79df2" (UID: "746cfe66-08c0-4eee-98c8-0eddb4d79df2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:33.749656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.749637    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-oauth-config" pod "746cfe66-08c0-4eee-98c8-0eddb4d79df2" (UID: "746cfe66-08c0-4eee-98c8-0eddb4d79df2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:18:33.749860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.749837    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746cfe66-08c0-4eee-98c8-0eddb4d79df2-kube-api-access-68qvv" pod "746cfe66-08c0-4eee-98c8-0eddb4d79df2" (UID: "746cfe66-08c0-4eee-98c8-0eddb4d79df2"). InnerVolumeSpecName "kube-api-access-68qvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:18:33.749860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.749845    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-serving-cert" pod "746cfe66-08c0-4eee-98c8-0eddb4d79df2" (UID: "746cfe66-08c0-4eee-98c8-0eddb4d79df2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:18:33.848428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.848369    5423 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-oauth-serving-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:33.848428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.848423    5423 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746cfe66-08c0-4eee-98c8-0eddb4d79df2-tmp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:33.848428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.848433    5423 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-oauth-config\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:33.848662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.848444    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qvv\" (UniqueName: \"kubernetes.io/projected/746cfe66-08c0-4eee-98c8-0eddb4d79df2-kube-api-access-68qvv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:33.848662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.848455    5423 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:33.848662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.848463    5423 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-config\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:33.848662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:33.848471    5423 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/746cfe66-08c0-4eee-98c8-0eddb4d79df2-console-serving-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:34.232948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.232866    5423 generic.go:356] "Generic (PLEG): container finished" podID="746cfe66-08c0-4eee-98c8-0eddb4d79df2" containerID="9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725" exitCode=0
Nov 01 14:18:34.232948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.232902    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bc7f65df-tb7mn" event={"ID":"746cfe66-08c0-4eee-98c8-0eddb4d79df2","Type":"ContainerDied","Data":"9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725"}
Nov 01 14:18:34.232948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.232925    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bc7f65df-tb7mn"
Nov 01 14:18:34.233175 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.232965    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bc7f65df-tb7mn" event={"ID":"746cfe66-08c0-4eee-98c8-0eddb4d79df2","Type":"ContainerDied","Data":"c04327d404d391d968dd789ade52288689695be972bc6a3806143a8ad2badddf"}
Nov 01 14:18:34.233175 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.232993    5423 scope.go:122] "RemoveContainer" containerID="9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725"
Nov 01 14:18:34.240776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.240760    5423 scope.go:122] "RemoveContainer" containerID="9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725"
Nov 01 14:18:34.241026 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:18:34.241007    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725\": container with ID starting with 9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725 not found: ID does not exist" containerID="9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725"
Nov 01 14:18:34.241078 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.241034    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725"} err="failed to get container status \"9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725\": rpc error: code = NotFound desc = could not find container \"9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725\": container with ID starting with 9774c4200f19a96c389050e8b6eaf93cf8d5e072f80441ae396850268d0bc725 not found: ID does not exist"
Nov 01 14:18:34.395159 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.395129    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65bc7f65df-tb7mn"]
Nov 01 14:18:34.445306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:34.445273    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65bc7f65df-tb7mn"]
Nov 01 14:18:35.146952 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:18:35.146913    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:18:35.957480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:35.957450    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746cfe66-08c0-4eee-98c8-0eddb4d79df2" path="/var/lib/kubelet/pods/746cfe66-08c0-4eee-98c8-0eddb4d79df2/volumes"
Nov 01 14:18:45.380265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.380229    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c765b696f-bbnvm" podUID="9b96d6ee-793c-4c5e-b02a-70841aca81ad" containerName="console" containerID="cri-o://b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d" gracePeriod=15
Nov 01 14:18:45.618715 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.618693    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:18:45.639709 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.639655    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-trusted-ca-bundle\") pod \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") "
Nov 01 14:18:45.639709 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.639701    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-serving-cert\") pod \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") "
Nov 01 14:18:45.639860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.639734    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-oauth-serving-cert\") pod \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") "
Nov 01 14:18:45.639860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.639760    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-oauth-config\") pod \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") "
Nov 01 14:18:45.639860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.639802    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b96d6ee-793c-4c5e-b02a-70841aca81ad-kube-api-access-sc8kv\" (UniqueName: \"kubernetes.io/projected/9b96d6ee-793c-4c5e-b02a-70841aca81ad-kube-api-access-sc8kv\") pod \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") "
Nov 01 14:18:45.639860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.639843    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-service-ca\") pod \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") "
Nov 01 14:18:45.640063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.639888    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-config\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-config\") pod \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") "
Nov 01 14:18:45.640063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.639929    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b96d6ee-793c-4c5e-b02a-70841aca81ad-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b96d6ee-793c-4c5e-b02a-70841aca81ad-tmp\") pod \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\" (UID: \"9b96d6ee-793c-4c5e-b02a-70841aca81ad\") "
Nov 01 14:18:45.640217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.640193    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-oauth-serving-cert" pod "9b96d6ee-793c-4c5e-b02a-70841aca81ad" (UID: "9b96d6ee-793c-4c5e-b02a-70841aca81ad"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:45.640334 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.640282    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b96d6ee-793c-4c5e-b02a-70841aca81ad-tmp" pod "9b96d6ee-793c-4c5e-b02a-70841aca81ad" (UID: "9b96d6ee-793c-4c5e-b02a-70841aca81ad"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:18:45.640334 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.640301    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-service-ca" pod "9b96d6ee-793c-4c5e-b02a-70841aca81ad" (UID: "9b96d6ee-793c-4c5e-b02a-70841aca81ad"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:45.640434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.640361    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-config" pod "9b96d6ee-793c-4c5e-b02a-70841aca81ad" (UID: "9b96d6ee-793c-4c5e-b02a-70841aca81ad"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:45.640434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.640377    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-trusted-ca-bundle" pod "9b96d6ee-793c-4c5e-b02a-70841aca81ad" (UID: "9b96d6ee-793c-4c5e-b02a-70841aca81ad"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:45.641846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.641829    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-serving-cert" pod "9b96d6ee-793c-4c5e-b02a-70841aca81ad" (UID: "9b96d6ee-793c-4c5e-b02a-70841aca81ad"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:18:45.642358 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.642330    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b96d6ee-793c-4c5e-b02a-70841aca81ad-kube-api-access-sc8kv" pod "9b96d6ee-793c-4c5e-b02a-70841aca81ad" (UID: "9b96d6ee-793c-4c5e-b02a-70841aca81ad"). InnerVolumeSpecName "kube-api-access-sc8kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:18:45.642358 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.642337    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-oauth-config" pod "9b96d6ee-793c-4c5e-b02a-70841aca81ad" (UID: "9b96d6ee-793c-4c5e-b02a-70841aca81ad"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:18:45.740930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.740897    5423 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-serving-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:45.740930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.740933    5423 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-oauth-serving-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:45.741099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.740951    5423 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-oauth-config\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:45.741099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.740966    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc8kv\" (UniqueName: \"kubernetes.io/projected/9b96d6ee-793c-4c5e-b02a-70841aca81ad-kube-api-access-sc8kv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:45.741099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.740981    5423 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:45.741099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.740997    5423 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-console-config\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:45.741099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.741011    5423 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b96d6ee-793c-4c5e-b02a-70841aca81ad-tmp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:45.741099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:45.741024    5423 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b96d6ee-793c-4c5e-b02a-70841aca81ad-trusted-ca-bundle\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:46.275151 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.275081    5423 generic.go:356] "Generic (PLEG): container finished" podID="9b96d6ee-793c-4c5e-b02a-70841aca81ad" containerID="b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d" exitCode=0
Nov 01 14:18:46.275151 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.275139    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c765b696f-bbnvm" event={"ID":"9b96d6ee-793c-4c5e-b02a-70841aca81ad","Type":"ContainerDied","Data":"b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d"}
Nov 01 14:18:46.275328 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.275177    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c765b696f-bbnvm" event={"ID":"9b96d6ee-793c-4c5e-b02a-70841aca81ad","Type":"ContainerDied","Data":"70b15ec53e663c52a2f239dd78eca6ee7002f1d92744f9bb14f0148fd0c81388"}
Nov 01 14:18:46.275328 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.275195    5423 scope.go:122] "RemoveContainer" containerID="b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d"
Nov 01 14:18:46.275328 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.275139    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c765b696f-bbnvm"
Nov 01 14:18:46.282704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.282689    5423 scope.go:122] "RemoveContainer" containerID="b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d"
Nov 01 14:18:46.282981 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:18:46.282957    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d\": container with ID starting with b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d not found: ID does not exist" containerID="b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d"
Nov 01 14:18:46.283094 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.282986    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d"} err="failed to get container status \"b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d\": rpc error: code = NotFound desc = could not find container \"b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d\": container with ID starting with b324e2cfd159c180ed07c4a124e55e4621ed636d9a0b4581b24ff4e581bf916d not found: ID does not exist"
Nov 01 14:18:46.437876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.437848    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c765b696f-bbnvm"]
Nov 01 14:18:46.488074 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:46.488050    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c765b696f-bbnvm"]
Nov 01 14:18:47.957055 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:18:47.957025    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b96d6ee-793c-4c5e-b02a-70841aca81ad" path="/var/lib/kubelet/pods/9b96d6ee-793c-4c5e-b02a-70841aca81ad/volumes"
Nov 01 14:19:05.148470 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:19:05.148432    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:19:16.450438 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.450410    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c5cc6f5d7-ltcdj"]
Nov 01 14:19:16.450903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.450729    5423 cpu_manager.go:414] "containerMap: removing container" podUID="746cfe66-08c0-4eee-98c8-0eddb4d79df2" containerName="console"
Nov 01 14:19:16.450903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.450740    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746cfe66-08c0-4eee-98c8-0eddb4d79df2" containerName="console"
Nov 01 14:19:16.450903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.450750    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9b96d6ee-793c-4c5e-b02a-70841aca81ad" containerName="console"
Nov 01 14:19:16.450903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.450755    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b96d6ee-793c-4c5e-b02a-70841aca81ad" containerName="console"
Nov 01 14:19:16.450903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.450824    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="746cfe66-08c0-4eee-98c8-0eddb4d79df2" containerName="console"
Nov 01 14:19:16.450903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.450832    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b96d6ee-793c-4c5e-b02a-70841aca81ad" containerName="console"
Nov 01 14:19:16.452778 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.452764    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.546283 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.546256    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5cc6f5d7-ltcdj"]
Nov 01 14:19:16.555944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.555924    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-serving-cert\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.556052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.555954    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-trusted-ca-bundle\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.556052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.555988    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-oauth-serving-cert\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.556124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.556083    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78d8\" (UniqueName: \"kubernetes.io/projected/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-kube-api-access-m78d8\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.556157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.556124    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-config\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.556190 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.556158    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-service-ca\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.556190 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.556174    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-tmp\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.556253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.556211    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-oauth-config\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657167 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657139    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-config\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657167 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657171    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-service-ca\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657333 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657188    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-tmp\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657333 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657319    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-oauth-config\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657362    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-serving-cert\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657515    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-trusted-ca-bundle\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657587    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-oauth-serving-cert\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657663 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657620    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78d8\" (UniqueName: \"kubernetes.io/projected/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-kube-api-access-m78d8\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657663 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657524    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-tmp\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.657977 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657954    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-service-ca\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.658073 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.657954    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-config\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.658229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.658209    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-trusted-ca-bundle\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.658266 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.658243    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-oauth-serving-cert\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.660344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.660317    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-oauth-config\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.660450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.660410    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-console-serving-cert\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.714470 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.714420    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78d8\" (UniqueName: \"kubernetes.io/projected/7767524e-1eb6-43c8-b53c-0dbf46ab81c5-kube-api-access-m78d8\") pod \"console-c5cc6f5d7-ltcdj\" (UID: \"7767524e-1eb6-43c8-b53c-0dbf46ab81c5\") " pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:16.763569 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:16.763541    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:17.173605 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:19:17.173564    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7767524e_1eb6_43c8_b53c_0dbf46ab81c5.slice/crio-af85b0480e879177c29268b669218c54e1032e3b3fab8117b2361b9426cc2815 WatchSource:0}: Error finding container af85b0480e879177c29268b669218c54e1032e3b3fab8117b2361b9426cc2815: Status 404 returned error can't find the container with id af85b0480e879177c29268b669218c54e1032e3b3fab8117b2361b9426cc2815
Nov 01 14:19:17.174476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:17.174456    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5cc6f5d7-ltcdj"]
Nov 01 14:19:17.384231 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:17.384194    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5cc6f5d7-ltcdj" event={"ID":"7767524e-1eb6-43c8-b53c-0dbf46ab81c5","Type":"ContainerStarted","Data":"e4d51990f436660713e9d9d50c3812efebd284c3974e8a13f6d352d66f798bdf"}
Nov 01 14:19:17.384447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:17.384251    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5cc6f5d7-ltcdj" event={"ID":"7767524e-1eb6-43c8-b53c-0dbf46ab81c5","Type":"ContainerStarted","Data":"af85b0480e879177c29268b669218c54e1032e3b3fab8117b2361b9426cc2815"}
Nov 01 14:19:17.499166 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:17.499081    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c5cc6f5d7-ltcdj" podStartSLOduration=1.499067809 podStartE2EDuration="1.499067809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:19:17.498856314 +0000 UTC m=+97.664341613" watchObservedRunningTime="2026-11-01 14:19:17.499067809 +0000 UTC m=+97.664553106"
Nov 01 14:19:26.763963 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:26.763877    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:26.764370 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:26.763964    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:26.768876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:26.768855    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:27.425115 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:27.425091    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c5cc6f5d7-ltcdj"
Nov 01 14:19:27.590241 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:27.590211    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58ccd5b85b-svg4v"]
Nov 01 14:19:35.146819 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:19:35.146760    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:19:52.612721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.612657    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58ccd5b85b-svg4v" podUID="369e3825-b4ec-4a0c-afaf-a58987feabdb" containerName="console" containerID="cri-o://11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8" gracePeriod=15
Nov 01 14:19:52.854102 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.854081    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:19:52.933494 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933428    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-config\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-config\") pod \"369e3825-b4ec-4a0c-afaf-a58987feabdb\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") "
Nov 01 14:19:52.933494 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933477    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-serving-cert\") pod \"369e3825-b4ec-4a0c-afaf-a58987feabdb\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") "
Nov 01 14:19:52.933695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933501    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-service-ca\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-service-ca\") pod \"369e3825-b4ec-4a0c-afaf-a58987feabdb\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") "
Nov 01 14:19:52.933695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933526    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369e3825-b4ec-4a0c-afaf-a58987feabdb-kube-api-access-ng55b\" (UniqueName: \"kubernetes.io/projected/369e3825-b4ec-4a0c-afaf-a58987feabdb-kube-api-access-ng55b\") pod \"369e3825-b4ec-4a0c-afaf-a58987feabdb\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") "
Nov 01 14:19:52.933695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933549    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-oauth-serving-cert\") pod \"369e3825-b4ec-4a0c-afaf-a58987feabdb\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") "
Nov 01 14:19:52.933695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933577    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369e3825-b4ec-4a0c-afaf-a58987feabdb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/369e3825-b4ec-4a0c-afaf-a58987feabdb-tmp\") pod \"369e3825-b4ec-4a0c-afaf-a58987feabdb\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") "
Nov 01 14:19:52.933695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933600    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-trusted-ca-bundle\") pod \"369e3825-b4ec-4a0c-afaf-a58987feabdb\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") "
Nov 01 14:19:52.933695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933669    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-oauth-config\") pod \"369e3825-b4ec-4a0c-afaf-a58987feabdb\" (UID: \"369e3825-b4ec-4a0c-afaf-a58987feabdb\") "
Nov 01 14:19:52.933984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933810    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-config" pod "369e3825-b4ec-4a0c-afaf-a58987feabdb" (UID: "369e3825-b4ec-4a0c-afaf-a58987feabdb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:19:52.933984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933924    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369e3825-b4ec-4a0c-afaf-a58987feabdb-tmp" pod "369e3825-b4ec-4a0c-afaf-a58987feabdb" (UID: "369e3825-b4ec-4a0c-afaf-a58987feabdb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:19:52.934059 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.933992    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-service-ca" pod "369e3825-b4ec-4a0c-afaf-a58987feabdb" (UID: "369e3825-b4ec-4a0c-afaf-a58987feabdb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:19:52.934176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.934152    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-trusted-ca-bundle" pod "369e3825-b4ec-4a0c-afaf-a58987feabdb" (UID: "369e3825-b4ec-4a0c-afaf-a58987feabdb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:19:52.934354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.934312    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-oauth-serving-cert" pod "369e3825-b4ec-4a0c-afaf-a58987feabdb" (UID: "369e3825-b4ec-4a0c-afaf-a58987feabdb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:19:52.935796 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.935775    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-oauth-config" pod "369e3825-b4ec-4a0c-afaf-a58987feabdb" (UID: "369e3825-b4ec-4a0c-afaf-a58987feabdb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:19:52.936245 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.936221    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-serving-cert" pod "369e3825-b4ec-4a0c-afaf-a58987feabdb" (UID: "369e3825-b4ec-4a0c-afaf-a58987feabdb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:19:52.936323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:52.936223    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369e3825-b4ec-4a0c-afaf-a58987feabdb-kube-api-access-ng55b" pod "369e3825-b4ec-4a0c-afaf-a58987feabdb" (UID: "369e3825-b4ec-4a0c-afaf-a58987feabdb"). InnerVolumeSpecName "kube-api-access-ng55b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:19:53.034157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.034129    5423 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369e3825-b4ec-4a0c-afaf-a58987feabdb-tmp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:19:53.034157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.034153    5423 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-trusted-ca-bundle\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:19:53.034313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.034162    5423 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-oauth-config\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:19:53.034313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.034172    5423 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-config\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:19:53.034313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.034181    5423 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/369e3825-b4ec-4a0c-afaf-a58987feabdb-console-serving-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:19:53.034313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.034190    5423 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:19:53.034313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.034199    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng55b\" (UniqueName: \"kubernetes.io/projected/369e3825-b4ec-4a0c-afaf-a58987feabdb-kube-api-access-ng55b\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:19:53.034313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.034208    5423 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/369e3825-b4ec-4a0c-afaf-a58987feabdb-oauth-serving-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:19:53.504886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.504850    5423 generic.go:356] "Generic (PLEG): container finished" podID="369e3825-b4ec-4a0c-afaf-a58987feabdb" containerID="11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8" exitCode=0
Nov 01 14:19:53.505075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.504893    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ccd5b85b-svg4v" event={"ID":"369e3825-b4ec-4a0c-afaf-a58987feabdb","Type":"ContainerDied","Data":"11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8"}
Nov 01 14:19:53.505075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.504903    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ccd5b85b-svg4v"
Nov 01 14:19:53.505075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.504931    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ccd5b85b-svg4v" event={"ID":"369e3825-b4ec-4a0c-afaf-a58987feabdb","Type":"ContainerDied","Data":"3d5dc5abb42aaad0fb15c27b720b5ea71cf344a7a722eb9fa5bcc5ee7ba98d53"}
Nov 01 14:19:53.505075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.504950    5423 scope.go:122] "RemoveContainer" containerID="11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8"
Nov 01 14:19:53.513074 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.513060    5423 scope.go:122] "RemoveContainer" containerID="11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8"
Nov 01 14:19:53.513296 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:19:53.513276    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8\": container with ID starting with 11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8 not found: ID does not exist" containerID="11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8"
Nov 01 14:19:53.513351 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.513302    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8"} err="failed to get container status \"11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8\": rpc error: code = NotFound desc = could not find container \"11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8\": container with ID starting with 11489e1609fe5eb611e3ba94df18b854788b6a1c77952b2206fd8cae84f11cd8 not found: ID does not exist"
Nov 01 14:19:53.667072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.667043    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58ccd5b85b-svg4v"]
Nov 01 14:19:53.717347 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.717318    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58ccd5b85b-svg4v"]
Nov 01 14:19:53.957605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:19:53.957578    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369e3825-b4ec-4a0c-afaf-a58987feabdb" path="/var/lib/kubelet/pods/369e3825-b4ec-4a0c-afaf-a58987feabdb/volumes"
Nov 01 14:20:05.147060 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:20:05.147022    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:20:35.147863 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:20:35.147827    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:21:05.147319 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:21:05.147283    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:21:20.267091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.267059    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"]
Nov 01 14:21:20.267609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.267312    5423 cpu_manager.go:414] "containerMap: removing container" podUID="369e3825-b4ec-4a0c-afaf-a58987feabdb" containerName="console"
Nov 01 14:21:20.267609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.267322    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369e3825-b4ec-4a0c-afaf-a58987feabdb" containerName="console"
Nov 01 14:21:20.267609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.267411    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="369e3825-b4ec-4a0c-afaf-a58987feabdb" containerName="console"
Nov 01 14:21:20.269219 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.269204    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.324446 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.324417    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"]
Nov 01 14:21:20.324631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.324613    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.324678 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.324643    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rhl4\" (UniqueName: \"kubernetes.io/projected/31b3f499-e6cd-4f1b-aac1-babc4397bda5-kube-api-access-2rhl4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.324738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.324723    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.373932 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.373901    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 14:21:20.373932 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.373916    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 14:21:20.391080 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.391053    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-dzln8\""
Nov 01 14:21:20.425109 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.425088    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.425208 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.425116    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rhl4\" (UniqueName: \"kubernetes.io/projected/31b3f499-e6cd-4f1b-aac1-babc4397bda5-kube-api-access-2rhl4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.425208 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.425151    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.425484 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.425469    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.425521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.425508    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.490100 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.490076    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rhl4\" (UniqueName: \"kubernetes.io/projected/31b3f499-e6cd-4f1b-aac1-babc4397bda5-kube-api-access-2rhl4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.493014 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.492993    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"]
Nov 01 14:21:20.495061 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.495042    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.547491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.547435    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 14:21:20.547491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.547462    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lmbst\""
Nov 01 14:21:20.553975 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.553951    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 14:21:20.554132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.554117    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 14:21:20.554204 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.554188    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 14:21:20.554965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.554947    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"]
Nov 01 14:21:20.581232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.581213    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:20.630415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.630359    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.630542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.630428    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lx8\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-kube-api-access-s7lx8\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.630542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.630489    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c630061-791a-4bc7-ad90-5912109efab2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.630542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.630534    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.630702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.630587    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.630702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.630612    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.630702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.630633    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.646102 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.646080    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 14:21:20.646197 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.646132    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 14:21:20.731572 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.731546    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c630061-791a-4bc7-ad90-5912109efab2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.731673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.731588    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.731673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.731622    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.731673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.731645    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.731673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.731660    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.731812 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.731684    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.731812 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.731706    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lx8\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-kube-api-access-s7lx8\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.732336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.732305    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.733961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.733936    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.734063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.733983    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c630061-791a-4bc7-ad90-5912109efab2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.734063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.734018    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.734143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.734092    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.788312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.788284    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lx8\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-kube-api-access-s7lx8\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.788530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.788514    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l46kz\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.805313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.805268    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:20.942861 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:21:20.942826    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b3f499_e6cd_4f1b_aac1_babc4397bda5.slice/crio-0ab91f42e330e313e72ed9fec0cd3ef5eb8875adb7eee136157459824620434b WatchSource:0}: Error finding container 0ab91f42e330e313e72ed9fec0cd3ef5eb8875adb7eee136157459824620434b: Status 404 returned error can't find the container with id 0ab91f42e330e313e72ed9fec0cd3ef5eb8875adb7eee136157459824620434b
Nov 01 14:21:20.942980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:20.942961    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"]
Nov 01 14:21:21.168040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:21.168006    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"]
Nov 01 14:21:21.168203 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:21:21.168181    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c630061_791a_4bc7_ad90_5912109efab2.slice/crio-14cae1aa7e21fc6b2435e869660af673768cbf9082369b93c32798f5e7d93f21 WatchSource:0}: Error finding container 14cae1aa7e21fc6b2435e869660af673768cbf9082369b93c32798f5e7d93f21: Status 404 returned error can't find the container with id 14cae1aa7e21fc6b2435e869660af673768cbf9082369b93c32798f5e7d93f21
Nov 01 14:21:21.797059 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:21.797022    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz" event={"ID":"4c630061-791a-4bc7-ad90-5912109efab2","Type":"ContainerStarted","Data":"14cae1aa7e21fc6b2435e869660af673768cbf9082369b93c32798f5e7d93f21"}
Nov 01 14:21:21.798108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:21.798084    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5" event={"ID":"31b3f499-e6cd-4f1b-aac1-babc4397bda5","Type":"ContainerStarted","Data":"0ab91f42e330e313e72ed9fec0cd3ef5eb8875adb7eee136157459824620434b"}
Nov 01 14:21:25.665935 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:25.665888    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:21:25.666314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:25.665969    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:21:25.816634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:25.816600    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz" event={"ID":"4c630061-791a-4bc7-ad90-5912109efab2","Type":"ContainerStarted","Data":"ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a"}
Nov 01 14:21:25.816796 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:25.816757    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:25.948084 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:25.947989    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz" podStartSLOduration=1.4529721 podStartE2EDuration="5.947969851s" totalImagesPullingTime="4.494997751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:21:25.946945692 +0000 UTC m=+226.112431002" watchObservedRunningTime="2026-11-01 14:21:25.947969851 +0000 UTC m=+226.113455151"
Nov 01 14:21:26.825199 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:26.825167    5423 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-l46kz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:21:26.825685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:26.825224    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz" podUID="4c630061-791a-4bc7-ad90-5912109efab2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:21:27.828850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:27.828822    5423 generic.go:356] "Generic (PLEG): container finished" podID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerID="62aceb85e06d51e0c994533b8309935446bed18876c513f25380454c4ccf6c7b" exitCode=0
Nov 01 14:21:27.829310 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:27.828906    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5" event={"ID":"31b3f499-e6cd-4f1b-aac1-babc4397bda5","Type":"ContainerDied","Data":"62aceb85e06d51e0c994533b8309935446bed18876c513f25380454c4ccf6c7b"}
Nov 01 14:21:29.825026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:29.824994    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:21:29.836685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:29.836653    5423 generic.go:356] "Generic (PLEG): container finished" podID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerID="7b6585645069498f7fd2eff0cec4a7d6ee0b019ec80776a34d8e137158e72e14" exitCode=0
Nov 01 14:21:29.836806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:29.836729    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5" event={"ID":"31b3f499-e6cd-4f1b-aac1-babc4397bda5","Type":"ContainerDied","Data":"7b6585645069498f7fd2eff0cec4a7d6ee0b019ec80776a34d8e137158e72e14"}
Nov 01 14:21:35.150703 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:21:35.150666    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:21:36.865420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:36.865327    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5" event={"ID":"31b3f499-e6cd-4f1b-aac1-babc4397bda5","Type":"ContainerStarted","Data":"06dc16d9c4f9fce5320ef1e6d83e2c80fbf1ef7c535b80ee409a59f2c252f8ce"}
Nov 01 14:21:37.870555 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:37.870521    5423 generic.go:356] "Generic (PLEG): container finished" podID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerID="06dc16d9c4f9fce5320ef1e6d83e2c80fbf1ef7c535b80ee409a59f2c252f8ce" exitCode=0
Nov 01 14:21:37.870941 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:37.870593    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5" event={"ID":"31b3f499-e6cd-4f1b-aac1-babc4397bda5","Type":"ContainerDied","Data":"06dc16d9c4f9fce5320ef1e6d83e2c80fbf1ef7c535b80ee409a59f2c252f8ce"}
Nov 01 14:21:38.991508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:38.991485    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:39.188037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.187949    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-util\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-util\") pod \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") "
Nov 01 14:21:39.188037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.187995    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-bundle\") pod \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") "
Nov 01 14:21:39.188037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.188017    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b3f499-e6cd-4f1b-aac1-babc4397bda5-kube-api-access-2rhl4\" (UniqueName: \"kubernetes.io/projected/31b3f499-e6cd-4f1b-aac1-babc4397bda5-kube-api-access-2rhl4\") pod \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\" (UID: \"31b3f499-e6cd-4f1b-aac1-babc4397bda5\") "
Nov 01 14:21:39.188598 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.188572    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-bundle" pod "31b3f499-e6cd-4f1b-aac1-babc4397bda5" (UID: "31b3f499-e6cd-4f1b-aac1-babc4397bda5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:21:39.190209 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.190188    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b3f499-e6cd-4f1b-aac1-babc4397bda5-kube-api-access-2rhl4" pod "31b3f499-e6cd-4f1b-aac1-babc4397bda5" (UID: "31b3f499-e6cd-4f1b-aac1-babc4397bda5"). InnerVolumeSpecName "kube-api-access-2rhl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:21:39.193161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.193124    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-util" pod "31b3f499-e6cd-4f1b-aac1-babc4397bda5" (UID: "31b3f499-e6cd-4f1b-aac1-babc4397bda5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:21:39.289305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.289271    5423 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-util\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:39.289305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.289297    5423 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b3f499-e6cd-4f1b-aac1-babc4397bda5-bundle\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:39.289305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.289307    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rhl4\" (UniqueName: \"kubernetes.io/projected/31b3f499-e6cd-4f1b-aac1-babc4397bda5-kube-api-access-2rhl4\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:39.878638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.878605    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5" event={"ID":"31b3f499-e6cd-4f1b-aac1-babc4397bda5","Type":"ContainerDied","Data":"0ab91f42e330e313e72ed9fec0cd3ef5eb8875adb7eee136157459824620434b"}
Nov 01 14:21:39.878800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.878640    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9cq5"
Nov 01 14:21:39.878800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:39.878649    5423 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ab91f42e330e313e72ed9fec0cd3ef5eb8875adb7eee136157459824620434b"
Nov 01 14:21:53.568130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.568086    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"]
Nov 01 14:21:53.568671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.568382    5423 cpu_manager.go:414] "containerMap: removing container" podUID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerName="util"
Nov 01 14:21:53.568671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.568405    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerName="util"
Nov 01 14:21:53.568671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.568415    5423 cpu_manager.go:414] "containerMap: removing container" podUID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerName="pull"
Nov 01 14:21:53.568671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.568420    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerName="pull"
Nov 01 14:21:53.568671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.568428    5423 cpu_manager.go:414] "containerMap: removing container" podUID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerName="extract"
Nov 01 14:21:53.568671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.568433    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerName="extract"
Nov 01 14:21:53.568671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.568496    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b3f499-e6cd-4f1b-aac1-babc4397bda5" containerName="extract"
Nov 01 14:21:53.570529 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.570506    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.586204 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.586176    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54r7\" (UniqueName: \"kubernetes.io/projected/e93e09fc-2484-4281-9f50-9c7eec5c7b81-kube-api-access-x54r7\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.586305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.586210    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e93e09fc-2484-4281-9f50-9c7eec5c7b81-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.586305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.586231    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.586305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.586278    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.586484 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.586422    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.586484 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.586474    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e93e09fc-2484-4281-9f50-9c7eec5c7b81-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.682473 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.682439    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 14:21:53.682637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.682575    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 14:21:53.682772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.682754    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 14:21:53.682841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.682788    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kgzw7\""
Nov 01 14:21:53.682841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.682786    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 14:21:53.682919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.682910    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 14:21:53.687234 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.687218    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e93e09fc-2484-4281-9f50-9c7eec5c7b81-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.687303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.687244    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.687303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.687261    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.687428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.687410    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.687513 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.687495    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e93e09fc-2484-4281-9f50-9c7eec5c7b81-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.687570 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.687533    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54r7\" (UniqueName: \"kubernetes.io/projected/e93e09fc-2484-4281-9f50-9c7eec5c7b81-kube-api-access-x54r7\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.687651 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.687628    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e93e09fc-2484-4281-9f50-9c7eec5c7b81-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.689611 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.689584    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e93e09fc-2484-4281-9f50-9c7eec5c7b81-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.689815 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.689796    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.689945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.689921    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.690001 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.689930    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e93e09fc-2484-4281-9f50-9c7eec5c7b81-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.700773 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.700749    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"]
Nov 01 14:21:53.751919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.751890    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54r7\" (UniqueName: \"kubernetes.io/projected/e93e09fc-2484-4281-9f50-9c7eec5c7b81-kube-api-access-x54r7\") pod \"opendatahub-operator-controller-manager-666cc8ff84-sb4d2\" (UID: \"e93e09fc-2484-4281-9f50-9c7eec5c7b81\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:53.883141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:53.883061    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:21:54.410974 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:21:54.410937    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93e09fc_2484_4281_9f50_9c7eec5c7b81.slice/crio-28cd4b21e1c400434ba8abc7acb076a84c219a0fa7736d4e9294944f4fc1e364 WatchSource:0}: Error finding container 28cd4b21e1c400434ba8abc7acb076a84c219a0fa7736d4e9294944f4fc1e364: Status 404 returned error can't find the container with id 28cd4b21e1c400434ba8abc7acb076a84c219a0fa7736d4e9294944f4fc1e364
Nov 01 14:21:54.411775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:54.411660    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"]
Nov 01 14:21:54.929876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:54.929843    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2" event={"ID":"e93e09fc-2484-4281-9f50-9c7eec5c7b81","Type":"ContainerStarted","Data":"28cd4b21e1c400434ba8abc7acb076a84c219a0fa7736d4e9294944f4fc1e364"}
Nov 01 14:21:58.945624 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:58.945592    5423 generic.go:356] "Generic (PLEG): container finished" podID="e93e09fc-2484-4281-9f50-9c7eec5c7b81" containerID="db5d8eec55a20a2dc846062b1fc9d68f413c87b2ac2b1772dee7ef48534ced79" exitCode=0
Nov 01 14:21:58.946076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:58.945681    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2" event={"ID":"e93e09fc-2484-4281-9f50-9c7eec5c7b81","Type":"ContainerDied","Data":"db5d8eec55a20a2dc846062b1fc9d68f413c87b2ac2b1772dee7ef48534ced79"}
Nov 01 14:21:59.950502 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:59.950472    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2" event={"ID":"e93e09fc-2484-4281-9f50-9c7eec5c7b81","Type":"ContainerStarted","Data":"bc0147c85b13e539b154bde815fe2223f657ac00c8240289ddfd98d1fd5e4def"}
Nov 01 14:21:59.950890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:21:59.950610    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:22:00.180530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:00.180442    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2" podStartSLOduration=2.371088449 podStartE2EDuration="7.180428128s" totalImagesPullingTime="4.403265667s" totalInitContainerRuntime="406.074012ms" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:00.129970124 +0000 UTC m=+260.295455422" watchObservedRunningTime="2026-11-01 14:22:00.180428128 +0000 UTC m=+260.345913424"
Nov 01 14:22:05.148020 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:22:05.147981    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:22:10.957191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:10.957157    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-sb4d2"
Nov 01 14:22:33.347352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.347279    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"]
Nov 01 14:22:33.353080 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.353063    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.408507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.408473    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-656df\""
Nov 01 14:22:33.459483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.459455    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"]
Nov 01 14:22:33.484047 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.484018    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096fc6d0-c03c-4cb9-b330-3a71744e4470-tls-certs\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.484197 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.484053    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdv76\" (UniqueName: \"kubernetes.io/projected/096fc6d0-c03c-4cb9-b330-3a71744e4470-kube-api-access-mdv76\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.484197 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.484074    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/096fc6d0-c03c-4cb9-b330-3a71744e4470-tmp\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.489529 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.489509    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 14:22:33.489626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.489613    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 14:22:33.585124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.585090    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096fc6d0-c03c-4cb9-b330-3a71744e4470-tls-certs\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.585280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.585133    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdv76\" (UniqueName: \"kubernetes.io/projected/096fc6d0-c03c-4cb9-b330-3a71744e4470-kube-api-access-mdv76\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.585280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.585156    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/096fc6d0-c03c-4cb9-b330-3a71744e4470-tmp\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.587452 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.587423    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/096fc6d0-c03c-4cb9-b330-3a71744e4470-tmp\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.587681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.587610    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096fc6d0-c03c-4cb9-b330-3a71744e4470-tls-certs\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.643208 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.643142    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdv76\" (UniqueName: \"kubernetes.io/projected/096fc6d0-c03c-4cb9-b330-3a71744e4470-kube-api-access-mdv76\") pod \"kube-auth-proxy-7fc9ccddb-v89s4\" (UID: \"096fc6d0-c03c-4cb9-b330-3a71744e4470\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:33.664060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:33.664031    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"
Nov 01 14:22:34.028475 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:22:34.028437    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096fc6d0_c03c_4cb9_b330_3a71744e4470.slice/crio-e03b1e3099b05bf88d61f60a68e0d467daccc65b16431ec255a7ca557e94ffea WatchSource:0}: Error finding container e03b1e3099b05bf88d61f60a68e0d467daccc65b16431ec255a7ca557e94ffea: Status 404 returned error can't find the container with id e03b1e3099b05bf88d61f60a68e0d467daccc65b16431ec255a7ca557e94ffea
Nov 01 14:22:34.029167 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:34.029144    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4"]
Nov 01 14:22:34.073969 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:34.073939    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4" event={"ID":"096fc6d0-c03c-4cb9-b330-3a71744e4470","Type":"ContainerStarted","Data":"e03b1e3099b05bf88d61f60a68e0d467daccc65b16431ec255a7ca557e94ffea"}
Nov 01 14:22:35.149231 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:22:35.149190    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:22:37.087550 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:37.087511    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4" event={"ID":"096fc6d0-c03c-4cb9-b330-3a71744e4470","Type":"ContainerStarted","Data":"45b9e076dd5a0b6846af913c588a2bb6ad37f8b62cd92a66b7e21b14789117e2"}
Nov 01 14:22:37.202561 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:37.202459    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-v89s4" podStartSLOduration=1.294870815 podStartE2EDuration="4.202444587s" totalImagesPullingTime="2.907573772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:37.201950387 +0000 UTC m=+297.367435686" watchObservedRunningTime="2026-11-01 14:22:37.202444587 +0000 UTC m=+297.367929885"
Nov 01 14:22:39.974366 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:39.974339    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:22:39.976239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:39.976219    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:22:40.171291 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.171259    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"]
Nov 01 14:22:40.174259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.174239    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.226127 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.226048    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"]
Nov 01 14:22:40.242313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.242283    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.242518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.242338    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.242518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.242357    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbzt\" (UniqueName: \"kubernetes.io/projected/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-kube-api-access-rvbzt\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.275433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.275404    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 01 14:22:40.275550 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.275440    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 01 14:22:40.276813 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.276800    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 14:22:40.276880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.276827    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 14:22:40.276924 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.276913    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jxs9f\""
Nov 01 14:22:40.343062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.343029    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.343239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.343067    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbzt\" (UniqueName: \"kubernetes.io/projected/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-kube-api-access-rvbzt\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.343239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.343132    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.343846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.343822    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.345584 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.345565    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.399966 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.399934    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbzt\" (UniqueName: \"kubernetes.io/projected/f521cd06-308b-48bb-a9a3-1ee4a82fbe04-kube-api-access-rvbzt\") pod \"kuadrant-console-plugin-66c4fb575d-55crp\" (UID: \"f521cd06-308b-48bb-a9a3-1ee4a82fbe04\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.498163 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.498089    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"
Nov 01 14:22:40.865503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:40.865472    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp"]
Nov 01 14:22:40.867032 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:22:40.867004    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf521cd06_308b_48bb_a9a3_1ee4a82fbe04.slice/crio-66639e6cb228a6b4c10de619b1243e35990c26842e7e536e3d7eaad040d24c71 WatchSource:0}: Error finding container 66639e6cb228a6b4c10de619b1243e35990c26842e7e536e3d7eaad040d24c71: Status 404 returned error can't find the container with id 66639e6cb228a6b4c10de619b1243e35990c26842e7e536e3d7eaad040d24c71
Nov 01 14:22:41.103704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:41.103668    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp" event={"ID":"f521cd06-308b-48bb-a9a3-1ee4a82fbe04","Type":"ContainerStarted","Data":"66639e6cb228a6b4c10de619b1243e35990c26842e7e536e3d7eaad040d24c71"}
Nov 01 14:22:42.963344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:42.962474    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:22:42.966241 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:42.966220    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:43.016999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.016969    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:22:43.065886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.065862    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 14:22:43.068319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.068288    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkfq8\" (UniqueName: \"kubernetes.io/projected/a4d27b75-03ea-410d-b323-86daa17409e9-kube-api-access-qkfq8\") pod \"limitador-limitador-7c9b87d574-dkfnx\" (UID: \"a4d27b75-03ea-410d-b323-86daa17409e9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:43.068514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.068491    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4d27b75-03ea-410d-b323-86daa17409e9-config-file\") pod \"limitador-limitador-7c9b87d574-dkfnx\" (UID: \"a4d27b75-03ea-410d-b323-86daa17409e9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:43.170129 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.169792    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4d27b75-03ea-410d-b323-86daa17409e9-config-file\") pod \"limitador-limitador-7c9b87d574-dkfnx\" (UID: \"a4d27b75-03ea-410d-b323-86daa17409e9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:43.170129 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.169846    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkfq8\" (UniqueName: \"kubernetes.io/projected/a4d27b75-03ea-410d-b323-86daa17409e9-kube-api-access-qkfq8\") pod \"limitador-limitador-7c9b87d574-dkfnx\" (UID: \"a4d27b75-03ea-410d-b323-86daa17409e9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:43.170649 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.170622    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4d27b75-03ea-410d-b323-86daa17409e9-config-file\") pod \"limitador-limitador-7c9b87d574-dkfnx\" (UID: \"a4d27b75-03ea-410d-b323-86daa17409e9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:43.227661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.227588    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkfq8\" (UniqueName: \"kubernetes.io/projected/a4d27b75-03ea-410d-b323-86daa17409e9-kube-api-access-qkfq8\") pod \"limitador-limitador-7c9b87d574-dkfnx\" (UID: \"a4d27b75-03ea-410d-b323-86daa17409e9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:43.263585 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.263558    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:22:43.284192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.284164    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:43.706322 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:22:43.706289    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d27b75_03ea_410d_b323_86daa17409e9.slice/crio-716fa7a0f66fb4222aab39411a6091a082e7f771e8f458aaeef2a195d97b5668 WatchSource:0}: Error finding container 716fa7a0f66fb4222aab39411a6091a082e7f771e8f458aaeef2a195d97b5668: Status 404 returned error can't find the container with id 716fa7a0f66fb4222aab39411a6091a082e7f771e8f458aaeef2a195d97b5668
Nov 01 14:22:43.706725 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.706702    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:22:43.708447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:43.708430    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:22:44.120139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:44.120066    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx" event={"ID":"a4d27b75-03ea-410d-b323-86daa17409e9","Type":"ContainerStarted","Data":"716fa7a0f66fb4222aab39411a6091a082e7f771e8f458aaeef2a195d97b5668"}
Nov 01 14:22:48.139957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:48.139916    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp" event={"ID":"f521cd06-308b-48bb-a9a3-1ee4a82fbe04","Type":"ContainerStarted","Data":"6d453db9b46a1d98c41d3b2f6dce44e6af44ebe2f37e0488aa7677bd5ade7d8f"}
Nov 01 14:22:48.141435 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:48.141409    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx" event={"ID":"a4d27b75-03ea-410d-b323-86daa17409e9","Type":"ContainerStarted","Data":"9fed0c80fcd79de8eb83f797fb4f7f95ba1ccd33cc1742153f2416dd8b2edde7"}
Nov 01 14:22:48.141569 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:48.141539    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:22:48.253803 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:48.253751    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-55crp" podStartSLOduration=2.008201371 podStartE2EDuration="8.253736835s" totalImagesPullingTime="6.245535464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:48.253233296 +0000 UTC m=+308.418718595" watchObservedRunningTime="2026-11-01 14:22:48.253736835 +0000 UTC m=+308.419222147"
Nov 01 14:22:48.366206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:48.366155    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx" podStartSLOduration=2.7131781029999997 podStartE2EDuration="6.366142143s" totalImagesPullingTime="3.65296404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:48.365814689 +0000 UTC m=+308.531300000" watchObservedRunningTime="2026-11-01 14:22:48.366142143 +0000 UTC m=+308.531627440"
Nov 01 14:22:59.146822 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:22:59.146788    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"
Nov 01 14:23:05.147090 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:23:05.147061    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:23:27.171365 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.171329    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-x48jt"]
Nov 01 14:23:27.182882 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.182854    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.228968 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.228932    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-x48jt"]
Nov 01 14:23:27.241289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.241263    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-metrics-certs\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.241471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.241299    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-webhook-cert\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.241547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.241486    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mb7\" (UniqueName: \"kubernetes.io/projected/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-kube-api-access-h9mb7\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.278374 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.278343    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 14:23:27.278374 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.278346    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 14:23:27.278565 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.278461    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lqgws\""
Nov 01 14:23:27.287035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.287017    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 14:23:27.287144 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.287089    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 14:23:27.287144 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.287130    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 14:23:27.342040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.342003    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-metrics-certs\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.342214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.342062    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-webhook-cert\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.342214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.342144    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mb7\" (UniqueName: \"kubernetes.io/projected/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-kube-api-access-h9mb7\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.344662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.344634    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-metrics-certs\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.344756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.344665    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-webhook-cert\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.398896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.398857    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mb7\" (UniqueName: \"kubernetes.io/projected/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-kube-api-access-h9mb7\") pod \"maas-controller-75bf7b5bf6-x48jt\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") " pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.495910 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.495834    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:27.904421 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:23:27.904382    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e87bc8c_e77e_42fe_ba31_b75c4a5967cc.slice/crio-f864a32444ed82b9f63c45974eec6e0e4d55b414f957a67fc0cd2c5331aed26f WatchSource:0}: Error finding container f864a32444ed82b9f63c45974eec6e0e4d55b414f957a67fc0cd2c5331aed26f: Status 404 returned error can't find the container with id f864a32444ed82b9f63c45974eec6e0e4d55b414f957a67fc0cd2c5331aed26f
Nov 01 14:23:27.905161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:27.905141    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-x48jt"]
Nov 01 14:23:28.295855 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:28.295772    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt" event={"ID":"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc","Type":"ContainerStarted","Data":"f864a32444ed82b9f63c45974eec6e0e4d55b414f957a67fc0cd2c5331aed26f"}
Nov 01 14:23:31.311160 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:31.311127    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt" event={"ID":"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc","Type":"ContainerStarted","Data":"8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe"}
Nov 01 14:23:31.311611 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:31.311259    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:23:31.426641 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:31.426588    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt" podStartSLOduration=1.471229469 podStartE2EDuration="4.426572414s" totalImagesPullingTime="2.955342945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:23:31.425810601 +0000 UTC m=+351.591295898" watchObservedRunningTime="2026-11-01 14:23:31.426572414 +0000 UTC m=+351.592057712"
Nov 01 14:23:35.148062 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:23:35.148023    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:23:42.316677 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:23:42.316644    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:24:00.943946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:00.943862    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"]
Nov 01 14:24:00.947313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:00.947295    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.003277 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.003241    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"]
Nov 01 14:24:01.132852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.132818    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.133044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.132860    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6tv\" (UniqueName: \"kubernetes.io/projected/3cb347b5-2346-4750-bf81-d005c9ae3fd9-kube-api-access-fr6tv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.133044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.132880    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.133044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.132939    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb347b5-2346-4750-bf81-d005c9ae3fd9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.133152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.133037    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.133152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.133115    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.133152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.133136    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.234331 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.234261    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.234331 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.234295    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb347b5-2346-4750-bf81-d005c9ae3fd9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.234331 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.234324    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.234603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.234362    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.234603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.234381    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.234603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.234435    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.234603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.234471    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6tv\" (UniqueName: \"kubernetes.io/projected/3cb347b5-2346-4750-bf81-d005c9ae3fd9-kube-api-access-fr6tv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.235281 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.235247    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.236980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.236934    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.237165 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.237148    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb347b5-2346-4750-bf81-d005c9ae3fd9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.237335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.237317    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.237441 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.237420    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3cb347b5-2346-4750-bf81-d005c9ae3fd9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.291640 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.291616    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb347b5-2346-4750-bf81-d005c9ae3fd9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.291745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.291647    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6tv\" (UniqueName: \"kubernetes.io/projected/3cb347b5-2346-4750-bf81-d005c9ae3fd9-kube-api-access-fr6tv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qk6ww\" (UID: \"3cb347b5-2346-4750-bf81-d005c9ae3fd9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.560250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.558543    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:01.992154 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:24:01.992112    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb347b5_2346_4750_bf81_d005c9ae3fd9.slice/crio-41803d7dc699442fcf9c47dde3816c618dc81c13bef0f8256629c52a25af1f15 WatchSource:0}: Error finding container 41803d7dc699442fcf9c47dde3816c618dc81c13bef0f8256629c52a25af1f15: Status 404 returned error can't find the container with id 41803d7dc699442fcf9c47dde3816c618dc81c13bef0f8256629c52a25af1f15
Nov 01 14:24:01.993258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.993231    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"]
Nov 01 14:24:01.994126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.994098    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:24:01.994176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:01.994152    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:24:02.427908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:02.427874    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww" event={"ID":"3cb347b5-2346-4750-bf81-d005c9ae3fd9","Type":"ContainerStarted","Data":"b169115f056445529356842cf476b2b8deceddb40ec59d6ec0c27c5c916f48a9"}
Nov 01 14:24:02.428095 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:02.427949    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww" event={"ID":"3cb347b5-2346-4750-bf81-d005c9ae3fd9","Type":"ContainerStarted","Data":"41803d7dc699442fcf9c47dde3816c618dc81c13bef0f8256629c52a25af1f15"}
Nov 01 14:24:02.428095 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:02.428005    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:02.649535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:02.649486    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww" podStartSLOduration=2.649473407 podStartE2EDuration="2.649473407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:02.59885062 +0000 UTC m=+382.764335915" watchObservedRunningTime="2026-11-01 14:24:02.649473407 +0000 UTC m=+382.814958704"
Nov 01 14:24:03.433782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:03.433743    5423 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qk6ww container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:24:03.434146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:03.433798    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww" podUID="3cb347b5-2346-4750-bf81-d005c9ae3fd9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:24:05.148582 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:24:05.148540    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:24:06.433468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:06.433441    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qk6ww"
Nov 01 14:24:06.651330 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:06.651299    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"]
Nov 01 14:24:06.651536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:06.651513    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz" podUID="4c630061-791a-4bc7-ad90-5912109efab2" containerName="discovery" containerID="cri-o://ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a" gracePeriod=30
Nov 01 14:24:06.904142 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:06.904113    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:24:07.083612 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.083586    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-cacerts\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-cacerts\") pod \"4c630061-791a-4bc7-ad90-5912109efab2\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") "
Nov 01 14:24:07.083786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.083622    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c630061-791a-4bc7-ad90-5912109efab2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c630061-791a-4bc7-ad90-5912109efab2-local-certs\") pod \"4c630061-791a-4bc7-ad90-5912109efab2\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") "
Nov 01 14:24:07.083786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.083656    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-kubeconfig\") pod \"4c630061-791a-4bc7-ad90-5912109efab2\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") "
Nov 01 14:24:07.083786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.083718    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-istio-token\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-istio-token\") pod \"4c630061-791a-4bc7-ad90-5912109efab2\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") "
Nov 01 14:24:07.083786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.083752    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-kube-api-access-s7lx8\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-kube-api-access-s7lx8\") pod \"4c630061-791a-4bc7-ad90-5912109efab2\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") "
Nov 01 14:24:07.084007 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.083786    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-dns-cert\") pod \"4c630061-791a-4bc7-ad90-5912109efab2\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") "
Nov 01 14:24:07.084007 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.083832    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-ca-configmap\") pod \"4c630061-791a-4bc7-ad90-5912109efab2\" (UID: \"4c630061-791a-4bc7-ad90-5912109efab2\") "
Nov 01 14:24:07.084311 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.084278    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-ca-configmap" pod "4c630061-791a-4bc7-ad90-5912109efab2" (UID: "4c630061-791a-4bc7-ad90-5912109efab2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:24:07.086056 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.086026    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-kubeconfig" pod "4c630061-791a-4bc7-ad90-5912109efab2" (UID: "4c630061-791a-4bc7-ad90-5912109efab2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:24:07.086164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.086052    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-cacerts" pod "4c630061-791a-4bc7-ad90-5912109efab2" (UID: "4c630061-791a-4bc7-ad90-5912109efab2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:24:07.086305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.086280    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-istio-token" pod "4c630061-791a-4bc7-ad90-5912109efab2" (UID: "4c630061-791a-4bc7-ad90-5912109efab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:24:07.086418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.086335    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-kube-api-access-s7lx8" pod "4c630061-791a-4bc7-ad90-5912109efab2" (UID: "4c630061-791a-4bc7-ad90-5912109efab2"). InnerVolumeSpecName "kube-api-access-s7lx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:24:07.086418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.086338    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c630061-791a-4bc7-ad90-5912109efab2-local-certs" pod "4c630061-791a-4bc7-ad90-5912109efab2" (UID: "4c630061-791a-4bc7-ad90-5912109efab2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:24:07.086547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.086530    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-dns-cert" pod "4c630061-791a-4bc7-ad90-5912109efab2" (UID: "4c630061-791a-4bc7-ad90-5912109efab2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:24:07.184623 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.184597    5423 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-kubeconfig\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:07.184623 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.184620    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:07.184623 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.184630    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7lx8\" (UniqueName: \"kubernetes.io/projected/4c630061-791a-4bc7-ad90-5912109efab2-kube-api-access-s7lx8\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:07.184623 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.184639    5423 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-dns-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:07.184870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.184648    5423 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c630061-791a-4bc7-ad90-5912109efab2-istio-csr-ca-configmap\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:07.184870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.184658    5423 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c630061-791a-4bc7-ad90-5912109efab2-cacerts\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:07.184870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.184667    5423 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c630061-791a-4bc7-ad90-5912109efab2-local-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:07.449096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.449007    5423 generic.go:356] "Generic (PLEG): container finished" podID="4c630061-791a-4bc7-ad90-5912109efab2" containerID="ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a" exitCode=0
Nov 01 14:24:07.449096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.449061    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"
Nov 01 14:24:07.449587 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.449109    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz" event={"ID":"4c630061-791a-4bc7-ad90-5912109efab2","Type":"ContainerDied","Data":"ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a"}
Nov 01 14:24:07.449587 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.449165    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz" event={"ID":"4c630061-791a-4bc7-ad90-5912109efab2","Type":"ContainerDied","Data":"14cae1aa7e21fc6b2435e869660af673768cbf9082369b93c32798f5e7d93f21"}
Nov 01 14:24:07.449587 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.449191    5423 scope.go:122] "RemoveContainer" containerID="ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a"
Nov 01 14:24:07.458378 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.458354    5423 scope.go:122] "RemoveContainer" containerID="ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a"
Nov 01 14:24:07.458678 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:24:07.458650    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a\": container with ID starting with ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a not found: ID does not exist" containerID="ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a"
Nov 01 14:24:07.458747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.458688    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a"} err="failed to get container status \"ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a\": rpc error: code = NotFound desc = could not find container \"ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a\": container with ID starting with ce0349b64d5fe46c19f08a64528369c415607c43a8b599b05a68ca397dd1a69a not found: ID does not exist"
Nov 01 14:24:07.611502 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.611469    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"]
Nov 01 14:24:07.661491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.661462    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l46kz"]
Nov 01 14:24:07.957786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:07.957754    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c630061-791a-4bc7-ad90-5912109efab2" path="/var/lib/kubelet/pods/4c630061-791a-4bc7-ad90-5912109efab2/volumes"
Nov 01 14:24:10.262476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:10.262443    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:24:15.610896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.610852    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tfc6t"]
Nov 01 14:24:15.611379 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.611321    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4c630061-791a-4bc7-ad90-5912109efab2" containerName="discovery"
Nov 01 14:24:15.611379 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.611339    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c630061-791a-4bc7-ad90-5912109efab2" containerName="discovery"
Nov 01 14:24:15.611544 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.611447    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c630061-791a-4bc7-ad90-5912109efab2" containerName="discovery"
Nov 01 14:24:15.615856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.615834    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t"
Nov 01 14:24:15.648961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.648935    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbz7\" (UniqueName: \"kubernetes.io/projected/51c6c036-b4f6-49bd-a8e5-294e442afe07-kube-api-access-8fbz7\") pod \"authorino-6f6944bcdd-tfc6t\" (UID: \"51c6c036-b4f6-49bd-a8e5-294e442afe07\") " pod="kuadrant-system/authorino-6f6944bcdd-tfc6t"
Nov 01 14:24:15.665616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.665589    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tfc6t"]
Nov 01 14:24:15.714687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.714664    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9sr64\""
Nov 01 14:24:15.749449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.749421    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbz7\" (UniqueName: \"kubernetes.io/projected/51c6c036-b4f6-49bd-a8e5-294e442afe07-kube-api-access-8fbz7\") pod \"authorino-6f6944bcdd-tfc6t\" (UID: \"51c6c036-b4f6-49bd-a8e5-294e442afe07\") " pod="kuadrant-system/authorino-6f6944bcdd-tfc6t"
Nov 01 14:24:15.794527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.794492    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"]
Nov 01 14:24:15.798117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.798094    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.807177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.807147    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbz7\" (UniqueName: \"kubernetes.io/projected/51c6c036-b4f6-49bd-a8e5-294e442afe07-kube-api-access-8fbz7\") pod \"authorino-6f6944bcdd-tfc6t\" (UID: \"51c6c036-b4f6-49bd-a8e5-294e442afe07\") " pod="kuadrant-system/authorino-6f6944bcdd-tfc6t"
Nov 01 14:24:15.849992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.849964    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53e3e14-ab21-43c8-b7d6-8cb588674e12-openshift-service-ca\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.850171 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.850026    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-maas-api-tls\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.850171 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.850142    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-metrics-certs\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.850273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.850212    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs89n\" (UniqueName: \"kubernetes.io/projected/c53e3e14-ab21-43c8-b7d6-8cb588674e12-kube-api-access-qs89n\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.851119 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.851094    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"]
Nov 01 14:24:15.854614 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.854594    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 14:24:15.854723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.854667    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7t7jm\""
Nov 01 14:24:15.854723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.854671    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 14:24:15.899710 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.899642    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 14:24:15.899710 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.899659    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 14:24:15.926664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.926638    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t"
Nov 01 14:24:15.947498 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.947471    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tfc6t"]
Nov 01 14:24:15.950570 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.950544    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-maas-api-tls\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.950662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.950585    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-metrics-certs\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.950662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.950620    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs89n\" (UniqueName: \"kubernetes.io/projected/c53e3e14-ab21-43c8-b7d6-8cb588674e12-kube-api-access-qs89n\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.950662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.950654    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53e3e14-ab21-43c8-b7d6-8cb588674e12-openshift-service-ca\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.951265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.951247    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53e3e14-ab21-43c8-b7d6-8cb588674e12-openshift-service-ca\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.953836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.953790    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-maas-api-tls\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:15.953960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:15.953908    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-metrics-certs\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:16.009523 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:16.009493    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs89n\" (UniqueName: \"kubernetes.io/projected/c53e3e14-ab21-43c8-b7d6-8cb588674e12-kube-api-access-qs89n\") pod \"maas-api-74dfdc6bb5-6f24x\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:16.111039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:16.111009    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:16.293932 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:24:16.293902    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51c6c036_b4f6_49bd_a8e5_294e442afe07.slice/crio-60d47e1e825344d96b3c8eb49e9c261918130196dfd7a17a90558c07fac86c74 WatchSource:0}: Error finding container 60d47e1e825344d96b3c8eb49e9c261918130196dfd7a17a90558c07fac86c74: Status 404 returned error can't find the container with id 60d47e1e825344d96b3c8eb49e9c261918130196dfd7a17a90558c07fac86c74
Nov 01 14:24:16.294778 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:16.294754    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tfc6t"]
Nov 01 14:24:16.473455 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:24:16.473424    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53e3e14_ab21_43c8_b7d6_8cb588674e12.slice/crio-a3b12983897e7e1990cb9e40525c5122e00fd132521d3a8c5c1c7c94911f44cc WatchSource:0}: Error finding container a3b12983897e7e1990cb9e40525c5122e00fd132521d3a8c5c1c7c94911f44cc: Status 404 returned error can't find the container with id a3b12983897e7e1990cb9e40525c5122e00fd132521d3a8c5c1c7c94911f44cc
Nov 01 14:24:16.474253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:16.474208    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"]
Nov 01 14:24:16.486458 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:16.486431    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t" event={"ID":"51c6c036-b4f6-49bd-a8e5-294e442afe07","Type":"ContainerStarted","Data":"60d47e1e825344d96b3c8eb49e9c261918130196dfd7a17a90558c07fac86c74"}
Nov 01 14:24:16.487361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:16.487341    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x" event={"ID":"c53e3e14-ab21-43c8-b7d6-8cb588674e12","Type":"ContainerStarted","Data":"a3b12983897e7e1990cb9e40525c5122e00fd132521d3a8c5c1c7c94911f44cc"}
Nov 01 14:24:19.501052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.500950    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x" event={"ID":"c53e3e14-ab21-43c8-b7d6-8cb588674e12","Type":"ContainerStarted","Data":"14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6"}
Nov 01 14:24:19.501052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.501023    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:19.502512 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.502484    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t" event={"ID":"51c6c036-b4f6-49bd-a8e5-294e442afe07","Type":"ContainerStarted","Data":"ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346"}
Nov 01 14:24:19.502616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.502579    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t" podUID="51c6c036-b4f6-49bd-a8e5-294e442afe07" containerName="authorino" containerID="cri-o://ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346" gracePeriod=30
Nov 01 14:24:19.617075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.617027    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x" podStartSLOduration=2.6133038280000003 podStartE2EDuration="4.617013158s" totalImagesPullingTime="2.00370933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:19.616341572 +0000 UTC m=+399.781826872" watchObservedRunningTime="2026-11-01 14:24:19.617013158 +0000 UTC m=+399.782498455"
Nov 01 14:24:19.728510 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.728459    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t" podStartSLOduration=1.965093633 podStartE2EDuration="4.72844598s" totalImagesPullingTime="2.763352347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:19.72798223 +0000 UTC m=+399.893467526" watchObservedRunningTime="2026-11-01 14:24:19.72844598 +0000 UTC m=+399.893931275"
Nov 01 14:24:19.748907 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.748887    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t"
Nov 01 14:24:19.786202 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.786146    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c6c036-b4f6-49bd-a8e5-294e442afe07-kube-api-access-8fbz7\" (UniqueName: \"kubernetes.io/projected/51c6c036-b4f6-49bd-a8e5-294e442afe07-kube-api-access-8fbz7\") pod \"51c6c036-b4f6-49bd-a8e5-294e442afe07\" (UID: \"51c6c036-b4f6-49bd-a8e5-294e442afe07\") "
Nov 01 14:24:19.788154 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.788128    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c6c036-b4f6-49bd-a8e5-294e442afe07-kube-api-access-8fbz7" pod "51c6c036-b4f6-49bd-a8e5-294e442afe07" (UID: "51c6c036-b4f6-49bd-a8e5-294e442afe07"). InnerVolumeSpecName "kube-api-access-8fbz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:24:19.887182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:19.887149    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fbz7\" (UniqueName: \"kubernetes.io/projected/51c6c036-b4f6-49bd-a8e5-294e442afe07-kube-api-access-8fbz7\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:20.507918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.507886    5423 generic.go:356] "Generic (PLEG): container finished" podID="51c6c036-b4f6-49bd-a8e5-294e442afe07" containerID="ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346" exitCode=0
Nov 01 14:24:20.508373 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.507934    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t"
Nov 01 14:24:20.508373 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.507959    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t" event={"ID":"51c6c036-b4f6-49bd-a8e5-294e442afe07","Type":"ContainerDied","Data":"ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346"}
Nov 01 14:24:20.508373 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.507999    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tfc6t" event={"ID":"51c6c036-b4f6-49bd-a8e5-294e442afe07","Type":"ContainerDied","Data":"60d47e1e825344d96b3c8eb49e9c261918130196dfd7a17a90558c07fac86c74"}
Nov 01 14:24:20.508373 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.508017    5423 scope.go:122] "RemoveContainer" containerID="ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346"
Nov 01 14:24:20.515921 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.515900    5423 scope.go:122] "RemoveContainer" containerID="ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346"
Nov 01 14:24:20.516170 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:24:20.516152    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346\": container with ID starting with ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346 not found: ID does not exist" containerID="ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346"
Nov 01 14:24:20.516218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.516178    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346"} err="failed to get container status \"ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346\": rpc error: code = NotFound desc = could not find container \"ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346\": container with ID starting with ca900c9df437845e771ec896e01059dd6b7d1a8400ca5aa60c1819b56fac3346 not found: ID does not exist"
Nov 01 14:24:20.668364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.668333    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tfc6t"]
Nov 01 14:24:20.718495 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:20.718469    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tfc6t"]
Nov 01 14:24:21.959272 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:21.959232    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c6c036-b4f6-49bd-a8e5-294e442afe07" path="/var/lib/kubelet/pods/51c6c036-b4f6-49bd-a8e5-294e442afe07/volumes"
Nov 01 14:24:25.514147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:24:25.514122    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 14:24:35.147560 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:24:35.147525    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:25:05.147560 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:25:05.147526    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:25:35.147335 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:25:35.147257    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:25:55.477925 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.477891    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lhbvt"]
Nov 01 14:25:55.478293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.478205    5423 cpu_manager.go:414] "containerMap: removing container" podUID="51c6c036-b4f6-49bd-a8e5-294e442afe07" containerName="authorino"
Nov 01 14:25:55.478293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.478217    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c6c036-b4f6-49bd-a8e5-294e442afe07" containerName="authorino"
Nov 01 14:25:55.478293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.478288    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c6c036-b4f6-49bd-a8e5-294e442afe07" containerName="authorino"
Nov 01 14:25:55.481049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.481024    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:55.533117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.533088    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lhbvt"]
Nov 01 14:25:55.561763 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.561738    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6kt\" (UniqueName: \"kubernetes.io/projected/2057931e-b603-4dec-b54e-ad98c5def8db-kube-api-access-jr6kt\") pod \"ai-gateway-controller-7d85675f98-lhbvt\" (UID: \"2057931e-b603-4dec-b54e-ad98c5def8db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:55.561880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.561787    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057931e-b603-4dec-b54e-ad98c5def8db-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-lhbvt\" (UID: \"2057931e-b603-4dec-b54e-ad98c5def8db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:55.581946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.581919    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 01 14:25:55.583454 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.583434    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-whksf\""
Nov 01 14:25:55.583551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.583462    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 01 14:25:55.662564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.662535    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057931e-b603-4dec-b54e-ad98c5def8db-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-lhbvt\" (UID: \"2057931e-b603-4dec-b54e-ad98c5def8db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:55.662735 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.662611    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6kt\" (UniqueName: \"kubernetes.io/projected/2057931e-b603-4dec-b54e-ad98c5def8db-kube-api-access-jr6kt\") pod \"ai-gateway-controller-7d85675f98-lhbvt\" (UID: \"2057931e-b603-4dec-b54e-ad98c5def8db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:55.665106 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.665082    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057931e-b603-4dec-b54e-ad98c5def8db-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-lhbvt\" (UID: \"2057931e-b603-4dec-b54e-ad98c5def8db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:55.728745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.728671    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr6kt\" (UniqueName: \"kubernetes.io/projected/2057931e-b603-4dec-b54e-ad98c5def8db-kube-api-access-jr6kt\") pod \"ai-gateway-controller-7d85675f98-lhbvt\" (UID: \"2057931e-b603-4dec-b54e-ad98c5def8db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:55.793214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:55.793187    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:56.167939 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:25:56.167908    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2057931e_b603_4dec_b54e_ad98c5def8db.slice/crio-e13530306abd802c81f77b022ba3cb4a224ed92740fd865a47ea3fdb08d26022 WatchSource:0}: Error finding container e13530306abd802c81f77b022ba3cb4a224ed92740fd865a47ea3fdb08d26022: Status 404 returned error can't find the container with id e13530306abd802c81f77b022ba3cb4a224ed92740fd865a47ea3fdb08d26022
Nov 01 14:25:56.168090 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:56.168069    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lhbvt"]
Nov 01 14:25:56.891989 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:56.891954    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt" event={"ID":"2057931e-b603-4dec-b54e-ad98c5def8db","Type":"ContainerStarted","Data":"e13530306abd802c81f77b022ba3cb4a224ed92740fd865a47ea3fdb08d26022"}
Nov 01 14:25:58.902088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:58.902051    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt" event={"ID":"2057931e-b603-4dec-b54e-ad98c5def8db","Type":"ContainerStarted","Data":"dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab"}
Nov 01 14:25:58.902564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:58.902130    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:25:59.015828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:25:59.015772    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt" podStartSLOduration=1.9031987849999998 podStartE2EDuration="4.015754591s" totalImagesPullingTime="2.112555806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:25:59.014944561 +0000 UTC m=+499.180429858" watchObservedRunningTime="2026-11-01 14:25:59.015754591 +0000 UTC m=+499.181239890"
Nov 01 14:26:00.914439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:00.914404    5423 generic.go:356] "Generic (PLEG): container finished" podID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerID="8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe" exitCode=1
Nov 01 14:26:00.914843 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:00.914476    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt" event={"ID":"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc","Type":"ContainerDied","Data":"8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe"}
Nov 01 14:26:00.914843 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:00.914803    5423 scope.go:122] "RemoveContainer" containerID="8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe"
Nov 01 14:26:01.920522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:01.920486    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt" event={"ID":"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc","Type":"ContainerStarted","Data":"ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb"}
Nov 01 14:26:01.920870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:01.920710    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:26:02.564750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.564714    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"]
Nov 01 14:26:02.567988 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.567968    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:02.620912 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.620889    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 14:26:02.623085 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.623061    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"]
Nov 01 14:26:02.718881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.718847    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ac5457-19e0-476a-9137-052ded3d5a13-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vvxtf\" (UID: \"f6ac5457-19e0-476a-9137-052ded3d5a13\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:02.719033 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.718953    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmfj\" (UniqueName: \"kubernetes.io/projected/f6ac5457-19e0-476a-9137-052ded3d5a13-kube-api-access-tmmfj\") pod \"payload-pre-processing-7b98bb678c-vvxtf\" (UID: \"f6ac5457-19e0-476a-9137-052ded3d5a13\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:02.724756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.724733    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"]
Nov 01 14:26:02.727961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.727945    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:02.780935 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.780894    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-h5hgn\""
Nov 01 14:26:02.781062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.780995    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"]
Nov 01 14:26:02.820372 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.820294    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m88v7\" (UniqueName: \"kubernetes.io/projected/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-kube-api-access-m88v7\") pod \"payload-processing-55fbbbf9dc-wc2s5\" (UID: \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:02.820517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.820403    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmfj\" (UniqueName: \"kubernetes.io/projected/f6ac5457-19e0-476a-9137-052ded3d5a13-kube-api-access-tmmfj\") pod \"payload-pre-processing-7b98bb678c-vvxtf\" (UID: \"f6ac5457-19e0-476a-9137-052ded3d5a13\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:02.820517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.820460    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ac5457-19e0-476a-9137-052ded3d5a13-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vvxtf\" (UID: \"f6ac5457-19e0-476a-9137-052ded3d5a13\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:02.820517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.820491    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wc2s5\" (UID: \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:02.821265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.821239    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ac5457-19e0-476a-9137-052ded3d5a13-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vvxtf\" (UID: \"f6ac5457-19e0-476a-9137-052ded3d5a13\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:02.877776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.877744    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmfj\" (UniqueName: \"kubernetes.io/projected/f6ac5457-19e0-476a-9137-052ded3d5a13-kube-api-access-tmmfj\") pod \"payload-pre-processing-7b98bb678c-vvxtf\" (UID: \"f6ac5457-19e0-476a-9137-052ded3d5a13\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:02.880580 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.880557    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:02.921240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.921212    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wc2s5\" (UID: \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:02.921697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.921269    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88v7\" (UniqueName: \"kubernetes.io/projected/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-kube-api-access-m88v7\") pod \"payload-processing-55fbbbf9dc-wc2s5\" (UID: \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:02.921912 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.921888    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wc2s5\" (UID: \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:02.978133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:02.978107    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m88v7\" (UniqueName: \"kubernetes.io/projected/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-kube-api-access-m88v7\") pod \"payload-processing-55fbbbf9dc-wc2s5\" (UID: \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:03.040166 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:03.040140    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:03.242831 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:26:03.242791    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ac5457_19e0_476a_9137_052ded3d5a13.slice/crio-02ff67e51e4372441355232683eeefd67f456da84b74ee34b16006ac410dd819 WatchSource:0}: Error finding container 02ff67e51e4372441355232683eeefd67f456da84b74ee34b16006ac410dd819: Status 404 returned error can't find the container with id 02ff67e51e4372441355232683eeefd67f456da84b74ee34b16006ac410dd819
Nov 01 14:26:03.243615 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:03.243588    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"]
Nov 01 14:26:03.402124 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:26:03.402083    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0162fda0_7a65_4c47_8aa7_46dd184f8cb5.slice/crio-7a5b51c1c467c55b0ee4dc2b29abbb40eda2b6e7cde014c49009884b163155d5 WatchSource:0}: Error finding container 7a5b51c1c467c55b0ee4dc2b29abbb40eda2b6e7cde014c49009884b163155d5: Status 404 returned error can't find the container with id 7a5b51c1c467c55b0ee4dc2b29abbb40eda2b6e7cde014c49009884b163155d5
Nov 01 14:26:03.403663 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:03.403642    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"]
Nov 01 14:26:03.929182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:03.929149    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf" event={"ID":"f6ac5457-19e0-476a-9137-052ded3d5a13","Type":"ContainerStarted","Data":"02ff67e51e4372441355232683eeefd67f456da84b74ee34b16006ac410dd819"}
Nov 01 14:26:03.930356 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:03.930336    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5" event={"ID":"0162fda0-7a65-4c47-8aa7-46dd184f8cb5","Type":"ContainerStarted","Data":"7a5b51c1c467c55b0ee4dc2b29abbb40eda2b6e7cde014c49009884b163155d5"}
Nov 01 14:26:05.154358 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:26:05.154315    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:26:06.947783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:06.947747    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf" event={"ID":"f6ac5457-19e0-476a-9137-052ded3d5a13","Type":"ContainerStarted","Data":"7f84b362d7f1ec1e29148959a8c7cd508726db0b23d4c8b47390145fc8155b4e"}
Nov 01 14:26:06.948252 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:06.947921    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:06.949271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:06.949246    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5" event={"ID":"0162fda0-7a65-4c47-8aa7-46dd184f8cb5","Type":"ContainerStarted","Data":"94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04"}
Nov 01 14:26:06.949405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:06.949373    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:07.061717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:07.061659    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf" podStartSLOduration=2.107558033 podStartE2EDuration="5.06164336s" totalImagesPullingTime="2.954085327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:07.060935004 +0000 UTC m=+507.226420301" watchObservedRunningTime="2026-11-01 14:26:07.06164336 +0000 UTC m=+507.227128658"
Nov 01 14:26:07.174222 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:07.174170    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5" podStartSLOduration=2.309090002 podStartE2EDuration="5.174155995s" totalImagesPullingTime="2.865065993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:07.172918402 +0000 UTC m=+507.338403700" watchObservedRunningTime="2026-11-01 14:26:07.174155995 +0000 UTC m=+507.339641292"
Nov 01 14:26:09.910705 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:09.910670    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:26:10.254327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.254242    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"]
Nov 01 14:26:10.257773 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.257752    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:10.291182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.291156    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v45z\" (UniqueName: \"kubernetes.io/projected/0afcba4a-013c-42bd-8216-b9b91a2c4bde-kube-api-access-2v45z\") pod \"ai-gateway-controller-9f778ddb6-jnqbk\" (UID: \"0afcba4a-013c-42bd-8216-b9b91a2c4bde\") " pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:10.291324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.291195    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afcba4a-013c-42bd-8216-b9b91a2c4bde-metrics-certs\") pod \"ai-gateway-controller-9f778ddb6-jnqbk\" (UID: \"0afcba4a-013c-42bd-8216-b9b91a2c4bde\") " pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:10.310495 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.310473    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"]
Nov 01 14:26:10.392045 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.392011    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v45z\" (UniqueName: \"kubernetes.io/projected/0afcba4a-013c-42bd-8216-b9b91a2c4bde-kube-api-access-2v45z\") pod \"ai-gateway-controller-9f778ddb6-jnqbk\" (UID: \"0afcba4a-013c-42bd-8216-b9b91a2c4bde\") " pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:10.392178 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.392059    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afcba4a-013c-42bd-8216-b9b91a2c4bde-metrics-certs\") pod \"ai-gateway-controller-9f778ddb6-jnqbk\" (UID: \"0afcba4a-013c-42bd-8216-b9b91a2c4bde\") " pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:10.394476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.394459    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afcba4a-013c-42bd-8216-b9b91a2c4bde-metrics-certs\") pod \"ai-gateway-controller-9f778ddb6-jnqbk\" (UID: \"0afcba4a-013c-42bd-8216-b9b91a2c4bde\") " pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:10.448988 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.448967    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v45z\" (UniqueName: \"kubernetes.io/projected/0afcba4a-013c-42bd-8216-b9b91a2c4bde-kube-api-access-2v45z\") pod \"ai-gateway-controller-9f778ddb6-jnqbk\" (UID: \"0afcba4a-013c-42bd-8216-b9b91a2c4bde\") " pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:10.570833 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.570807    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:10.930148 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:26:10.930118    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afcba4a_013c_42bd_8216_b9b91a2c4bde.slice/crio-5ed911e92d3175e41fa86df0ccda688c42665942bc7d46ddb65af456c40d3723 WatchSource:0}: Error finding container 5ed911e92d3175e41fa86df0ccda688c42665942bc7d46ddb65af456c40d3723: Status 404 returned error can't find the container with id 5ed911e92d3175e41fa86df0ccda688c42665942bc7d46ddb65af456c40d3723
Nov 01 14:26:10.931183 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.931147    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"]
Nov 01 14:26:10.964556 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:10.964525    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk" event={"ID":"0afcba4a-013c-42bd-8216-b9b91a2c4bde","Type":"ContainerStarted","Data":"5ed911e92d3175e41fa86df0ccda688c42665942bc7d46ddb65af456c40d3723"}
Nov 01 14:26:11.970493 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:11.970459    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk" event={"ID":"0afcba4a-013c-42bd-8216-b9b91a2c4bde","Type":"ContainerStarted","Data":"345fb2e04487fd7d80fdefaec05057da944850f7693ac399a571871491c93d73"}
Nov 01 14:26:11.970945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:11.970605    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:12.084447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:12.084344    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk" podStartSLOduration=1.1985203819999999 podStartE2EDuration="2.084328738s" totalImagesPullingTime="885.808356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:12.083534217 +0000 UTC m=+512.249019514" watchObservedRunningTime="2026-11-01 14:26:12.084328738 +0000 UTC m=+512.249814029"
Nov 01 14:26:12.926888 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:12.926850    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 14:26:17.959936 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:17.959911    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 14:26:17.960343 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:17.959963    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 14:26:22.977413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:22.977369    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-9f778ddb6-jnqbk"
Nov 01 14:26:23.141943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.141912    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lhbvt"]
Nov 01 14:26:23.142144 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.142108    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt" podUID="2057931e-b603-4dec-b54e-ad98c5def8db" containerName="manager" containerID="cri-o://dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab" gracePeriod=30
Nov 01 14:26:23.377826 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.377805    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:26:23.505849 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.505820    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2057931e-b603-4dec-b54e-ad98c5def8db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057931e-b603-4dec-b54e-ad98c5def8db-metrics-certs\") pod \"2057931e-b603-4dec-b54e-ad98c5def8db\" (UID: \"2057931e-b603-4dec-b54e-ad98c5def8db\") "
Nov 01 14:26:23.506013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.505872    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2057931e-b603-4dec-b54e-ad98c5def8db-kube-api-access-jr6kt\" (UniqueName: \"kubernetes.io/projected/2057931e-b603-4dec-b54e-ad98c5def8db-kube-api-access-jr6kt\") pod \"2057931e-b603-4dec-b54e-ad98c5def8db\" (UID: \"2057931e-b603-4dec-b54e-ad98c5def8db\") "
Nov 01 14:26:23.508160 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.508085    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2057931e-b603-4dec-b54e-ad98c5def8db-metrics-certs" pod "2057931e-b603-4dec-b54e-ad98c5def8db" (UID: "2057931e-b603-4dec-b54e-ad98c5def8db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:26:23.508160 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.508102    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2057931e-b603-4dec-b54e-ad98c5def8db-kube-api-access-jr6kt" pod "2057931e-b603-4dec-b54e-ad98c5def8db" (UID: "2057931e-b603-4dec-b54e-ad98c5def8db"). InnerVolumeSpecName "kube-api-access-jr6kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:26:23.607039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.607015    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057931e-b603-4dec-b54e-ad98c5def8db-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:26:23.607149 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:23.607040    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr6kt\" (UniqueName: \"kubernetes.io/projected/2057931e-b603-4dec-b54e-ad98c5def8db-kube-api-access-jr6kt\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:26:24.026614 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.026576    5423 generic.go:356] "Generic (PLEG): container finished" podID="2057931e-b603-4dec-b54e-ad98c5def8db" containerID="dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab" exitCode=0
Nov 01 14:26:24.027008 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.026627    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt"
Nov 01 14:26:24.027008 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.026647    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt" event={"ID":"2057931e-b603-4dec-b54e-ad98c5def8db","Type":"ContainerDied","Data":"dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab"}
Nov 01 14:26:24.027008 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.026700    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-lhbvt" event={"ID":"2057931e-b603-4dec-b54e-ad98c5def8db","Type":"ContainerDied","Data":"e13530306abd802c81f77b022ba3cb4a224ed92740fd865a47ea3fdb08d26022"}
Nov 01 14:26:24.027008 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.026728    5423 scope.go:122] "RemoveContainer" containerID="dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab"
Nov 01 14:26:24.034957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.034935    5423 scope.go:122] "RemoveContainer" containerID="dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab"
Nov 01 14:26:24.035225 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:26:24.035200    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab\": container with ID starting with dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab not found: ID does not exist" containerID="dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab"
Nov 01 14:26:24.035320 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.035232    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab"} err="failed to get container status \"dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab\": rpc error: code = NotFound desc = could not find container \"dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab\": container with ID starting with dff3dc5b6fcc8d7d068b4ae65115f8baa0c80317dca96cc60034ce09c18e58ab not found: ID does not exist"
Nov 01 14:26:24.188352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.188323    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lhbvt"]
Nov 01 14:26:24.238266 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:24.238192    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lhbvt"]
Nov 01 14:26:25.958252 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:25.958215    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2057931e-b603-4dec-b54e-ad98c5def8db" path="/var/lib/kubelet/pods/2057931e-b603-4dec-b54e-ad98c5def8db/volumes"
Nov 01 14:26:32.397652 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.397617    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"]
Nov 01 14:26:32.400917 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.397988    5423 cpu_manager.go:414] "containerMap: removing container" podUID="2057931e-b603-4dec-b54e-ad98c5def8db" containerName="manager"
Nov 01 14:26:32.400917 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.398001    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2057931e-b603-4dec-b54e-ad98c5def8db" containerName="manager"
Nov 01 14:26:32.400917 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.398082    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="2057931e-b603-4dec-b54e-ad98c5def8db" containerName="manager"
Nov 01 14:26:32.402209 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.402193    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.453808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.453782    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"]
Nov 01 14:26:32.483946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.483916    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.484060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.483953    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.484060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.483976    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4bq\" (UniqueName: \"kubernetes.io/projected/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-kube-api-access-pw4bq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.484131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.484085    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.484169 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.484159    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.484205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.484181    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.502666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.502644    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 14:26:32.502922 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.502905    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 14:26:32.504555 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.504540    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 14:26:32.504600 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.504547    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cbdjr\""
Nov 01 14:26:32.584788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.584757    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4bq\" (UniqueName: \"kubernetes.io/projected/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-kube-api-access-pw4bq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.584937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.584796    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.584937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.584844    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.584937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.584871    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.584937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.584925    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.585138 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.584963    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.585376 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.585349    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.585376 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.585374    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.585516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.585418    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.587050 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.587029    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.587297 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.587281    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.641788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.641762    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4bq\" (UniqueName: \"kubernetes.io/projected/635d4701-f7ba-4bbc-81bc-352f89a2f4f3-kube-api-access-pw4bq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z\" (UID: \"635d4701-f7ba-4bbc-81bc-352f89a2f4f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:32.715123 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:32.715061    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:33.080164 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:26:33.080138    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635d4701_f7ba_4bbc_81bc_352f89a2f4f3.slice/crio-2ae286d78ebbc72333dd626ee4fba7949aeb69e4ebca2dff985da2d9616a2338 WatchSource:0}: Error finding container 2ae286d78ebbc72333dd626ee4fba7949aeb69e4ebca2dff985da2d9616a2338: Status 404 returned error can't find the container with id 2ae286d78ebbc72333dd626ee4fba7949aeb69e4ebca2dff985da2d9616a2338
Nov 01 14:26:33.081022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:33.081001    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"]
Nov 01 14:26:34.072890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:34.072820    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z" event={"ID":"635d4701-f7ba-4bbc-81bc-352f89a2f4f3","Type":"ContainerStarted","Data":"2ae286d78ebbc72333dd626ee4fba7949aeb69e4ebca2dff985da2d9616a2338"}
Nov 01 14:26:35.150433 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:26:35.150376    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:26:35.963744 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:35.963710    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:26:36.084206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:36.084173    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z" event={"ID":"635d4701-f7ba-4bbc-81bc-352f89a2f4f3","Type":"ContainerStarted","Data":"e764139c033ae1fe6e5894fea0a1b212e93a93a5626b02e69945ca8c4377af5e"}
Nov 01 14:26:36.199290 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:36.199238    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z" podStartSLOduration=1.975568926 podStartE2EDuration="4.199222216s" totalImagesPullingTime="2.22365329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:36.198401064 +0000 UTC m=+536.363886358" watchObservedRunningTime="2026-11-01 14:26:36.199222216 +0000 UTC m=+536.364707508"
Nov 01 14:26:36.716172 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:36.716137    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:42.716060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:42.716021    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:42.728419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:42.728376    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:43.124338 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:43.124311    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z"
Nov 01 14:26:44.231815 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.231783    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"]
Nov 01 14:26:44.238735 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.238713    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.287687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.287658    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"]
Nov 01 14:26:44.291054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.291027    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 14:26:44.383063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.383028    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.383221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.383071    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.383221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.383176    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946e8442-e532-48b6-a5ad-ef769fb2e71c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.383294 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.383218    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.383294 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.383239    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvg5\" (UniqueName: \"kubernetes.io/projected/946e8442-e532-48b6-a5ad-ef769fb2e71c-kube-api-access-7vvg5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.383364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.383324    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.484599 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.484520    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946e8442-e532-48b6-a5ad-ef769fb2e71c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.484599 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.484578    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.484806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.484605    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvg5\" (UniqueName: \"kubernetes.io/projected/946e8442-e532-48b6-a5ad-ef769fb2e71c-kube-api-access-7vvg5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.484806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.484648    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.484806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.484684    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.484806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.484717    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.485126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.485101    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.485126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.485127    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.485318 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.485185    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.486946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.486927    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946e8442-e532-48b6-a5ad-ef769fb2e71c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.487117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.487102    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946e8442-e532-48b6-a5ad-ef769fb2e71c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.541020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.540991    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvg5\" (UniqueName: \"kubernetes.io/projected/946e8442-e532-48b6-a5ad-ef769fb2e71c-kube-api-access-7vvg5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hxdbf\" (UID: \"946e8442-e532-48b6-a5ad-ef769fb2e71c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.551855 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.551833    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:44.850535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.850498    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:26:44.954232 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:26:44.954201    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946e8442_e532_48b6_a5ad_ef769fb2e71c.slice/crio-64eaaacf8c3c043a7e74dbc4021d1526081fbb7a101b25daa586cfd408a074b7 WatchSource:0}: Error finding container 64eaaacf8c3c043a7e74dbc4021d1526081fbb7a101b25daa586cfd408a074b7: Status 404 returned error can't find the container with id 64eaaacf8c3c043a7e74dbc4021d1526081fbb7a101b25daa586cfd408a074b7
Nov 01 14:26:44.955202 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:44.955180    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"]
Nov 01 14:26:45.128967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:45.128883    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf" event={"ID":"946e8442-e532-48b6-a5ad-ef769fb2e71c","Type":"ContainerStarted","Data":"64eaaacf8c3c043a7e74dbc4021d1526081fbb7a101b25daa586cfd408a074b7"}
Nov 01 14:26:46.134647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:46.134608    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf" event={"ID":"946e8442-e532-48b6-a5ad-ef769fb2e71c","Type":"ContainerStarted","Data":"8e2868bf76b74bd610f96c0e79f96511fdd5b66cb2f7213ac499f0d028c5c7fc"}
Nov 01 14:26:46.248951 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:46.248907    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf" podStartSLOduration=1.6409150289999999 podStartE2EDuration="2.248895976s" totalImagesPullingTime="607.980947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:46.248182731 +0000 UTC m=+546.413668029" watchObservedRunningTime="2026-11-01 14:26:46.248895976 +0000 UTC m=+546.414381273"
Nov 01 14:26:46.552363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:46.552293    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:51.031401 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.031303    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"]
Nov 01 14:26:51.035012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.034989    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.088068 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.088036    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"]
Nov 01 14:26:51.136726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.136700    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 14:26:51.142044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.142017    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.142158 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.142069    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.142198 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.142160    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.142198 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.142186    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.142265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.142222    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beffa1b9-f1df-4269-a237-322aaecf3c18-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.142324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.142303    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrrv\" (UniqueName: \"kubernetes.io/projected/beffa1b9-f1df-4269-a237-322aaecf3c18-kube-api-access-5vrrv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.242993    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.243074    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.243118    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.243260    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.243364    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beffa1b9-f1df-4269-a237-322aaecf3c18-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.243402    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.243460    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.243515    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.243911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.243878    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrrv\" (UniqueName: \"kubernetes.io/projected/beffa1b9-f1df-4269-a237-322aaecf3c18-kube-api-access-5vrrv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.245540 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.245507    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beffa1b9-f1df-4269-a237-322aaecf3c18-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.245795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.245773    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beffa1b9-f1df-4269-a237-322aaecf3c18-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.300431 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.300366    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrrv\" (UniqueName: \"kubernetes.io/projected/beffa1b9-f1df-4269-a237-322aaecf3c18-kube-api-access-5vrrv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv\" (UID: \"beffa1b9-f1df-4269-a237-322aaecf3c18\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.347824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.347793    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:51.715907 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:51.715873    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"]
Nov 01 14:26:51.716903 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:26:51.716879    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeffa1b9_f1df_4269_a237_322aaecf3c18.slice/crio-2610fefc2042765a4ecf1d3e34e3ba962c8c37547cedd0bac919bb31d8888375 WatchSource:0}: Error finding container 2610fefc2042765a4ecf1d3e34e3ba962c8c37547cedd0bac919bb31d8888375: Status 404 returned error can't find the container with id 2610fefc2042765a4ecf1d3e34e3ba962c8c37547cedd0bac919bb31d8888375
Nov 01 14:26:52.160716 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:52.160678    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv" event={"ID":"beffa1b9-f1df-4269-a237-322aaecf3c18","Type":"ContainerStarted","Data":"2610fefc2042765a4ecf1d3e34e3ba962c8c37547cedd0bac919bb31d8888375"}
Nov 01 14:26:53.167316 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:53.167280    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv" event={"ID":"beffa1b9-f1df-4269-a237-322aaecf3c18","Type":"ContainerStarted","Data":"5615c1cef547374971c33f0b3ad6625d9305a0a018be5d1eb1d9d4b11dff0036"}
Nov 01 14:26:53.282139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:53.282086    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv" podStartSLOduration=2.680070894 podStartE2EDuration="3.282072008s" totalImagesPullingTime="602.001114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:53.281246785 +0000 UTC m=+553.446732082" watchObservedRunningTime="2026-11-01 14:26:53.282072008 +0000 UTC m=+553.447557306"
Nov 01 14:26:53.348643 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:53.348607    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:26:54.552990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:54.552951    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:54.565337 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:54.565313    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:55.191284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:55.191253    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hxdbf"
Nov 01 14:26:55.351522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:55.351493    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:26:58.478679 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.478650    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"]
Nov 01 14:26:58.482156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.482138    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.496219 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.496195    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.496322 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.496227    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.496322 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.496247    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.496413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.496364    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.496413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.496381    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhpd\" (UniqueName: \"kubernetes.io/projected/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-kube-api-access-hrhpd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.496521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.496508    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.584586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.584560    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 14:26:58.585870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.585849    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"]
Nov 01 14:26:58.596904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.596874    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.597048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.596906    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhpd\" (UniqueName: \"kubernetes.io/projected/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-kube-api-access-hrhpd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.597048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.596963    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.597164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.597097    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.597164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.597130    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.597164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.597153    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.597505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.597479    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.597608 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.597531    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.597608 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.597553    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.599312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.599289    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.599592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.599576    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.654568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.654537    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhpd\" (UniqueName: \"kubernetes.io/projected/ad617adb-5b9f-4927-ad5e-b8083aa5d1c2-kube-api-access-hrhpd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk\" (UID: \"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:58.795508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:58.795419    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:26:59.159326 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:26:59.159289    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad617adb_5b9f_4927_ad5e_b8083aa5d1c2.slice/crio-f12ad84563e5b6672ae47f10e91fe840a9195d940aa86ebce1d5212a25d48ca0 WatchSource:0}: Error finding container f12ad84563e5b6672ae47f10e91fe840a9195d940aa86ebce1d5212a25d48ca0: Status 404 returned error can't find the container with id f12ad84563e5b6672ae47f10e91fe840a9195d940aa86ebce1d5212a25d48ca0
Nov 01 14:26:59.160357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:59.160336    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"]
Nov 01 14:26:59.193747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:26:59.193716    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk" event={"ID":"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2","Type":"ContainerStarted","Data":"f12ad84563e5b6672ae47f10e91fe840a9195d940aa86ebce1d5212a25d48ca0"}
Nov 01 14:27:00.199965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:00.199925    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk" event={"ID":"ad617adb-5b9f-4927-ad5e-b8083aa5d1c2","Type":"ContainerStarted","Data":"f66613a45b2150b4cadb242b398c775748b2978795bbf5f226aa2cb9a3c82c3e"}
Nov 01 14:27:00.314023 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:00.313967    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk" podStartSLOduration=1.750307774 podStartE2EDuration="2.313955086s" totalImagesPullingTime="563.647312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:27:00.313823781 +0000 UTC m=+560.479309079" watchObservedRunningTime="2026-11-01 14:27:00.313955086 +0000 UTC m=+560.479440384"
Nov 01 14:27:00.796348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:00.796315    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:27:01.348525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:01.348496    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:27:01.360930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:01.360909    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:27:02.219263 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:02.219234    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv"
Nov 01 14:27:05.149271 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:27:05.149235    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:27:08.481177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:08.481145    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:27:08.796023 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:08.795939    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:27:08.808585 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:08.808559    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:27:09.247810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:09.247783    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk"
Nov 01 14:27:14.892330 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.892301    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"]
Nov 01 14:27:14.897576 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.897560    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927421    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927468    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2f92\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-kube-api-access-x2f92\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927544    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927593    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927618    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260e919a-8650-4d1f-8023-9f41eaba3278-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927809 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927688    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260e919a-8650-4d1f-8023-9f41eaba3278-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927809 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927713    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927865    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:14.927940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:14.927905    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.000098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.000060    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58vzm\""
Nov 01 14:27:15.001511 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.001490    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"]
Nov 01 14:27:15.028948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.028921    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029046 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.028952    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2f92\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-kube-api-access-x2f92\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029046 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.028975    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029046 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029004    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029373    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029464    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029147    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260e919a-8650-4d1f-8023-9f41eaba3278-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029812 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029793    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260e919a-8650-4d1f-8023-9f41eaba3278-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029811    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260e919a-8650-4d1f-8023-9f41eaba3278-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029826    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029973 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029933    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.029973 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.029969    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.030272 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.030250    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.030336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.030293    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.031441 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.031420    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.032637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.032620    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260e919a-8650-4d1f-8023-9f41eaba3278-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.085192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.085163    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2f92\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-kube-api-access-x2f92\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.085719 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.085701    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-drf59\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.210325 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.210270    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:15.622470 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:27:15.622430    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260e919a_8650_4d1f_8023_9f41eaba3278.slice/crio-3e8789bb912c550e44087386928a452cd52bd2e2ad9cc43712350c65cf40e9b1 WatchSource:0}: Error finding container 3e8789bb912c550e44087386928a452cd52bd2e2ad9cc43712350c65cf40e9b1: Status 404 returned error can't find the container with id 3e8789bb912c550e44087386928a452cd52bd2e2ad9cc43712350c65cf40e9b1
Nov 01 14:27:15.623898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:15.623871    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"]
Nov 01 14:27:16.269092 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:16.269060    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" event={"ID":"260e919a-8650-4d1f-8023-9f41eaba3278","Type":"ContainerStarted","Data":"3e8789bb912c550e44087386928a452cd52bd2e2ad9cc43712350c65cf40e9b1"}
Nov 01 14:27:19.712290 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:19.712258    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:27:19.712598 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:19.712339    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:27:19.712598 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:19.712370    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:27:20.289075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:20.289036    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" event={"ID":"260e919a-8650-4d1f-8023-9f41eaba3278","Type":"ContainerStarted","Data":"fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c"}
Nov 01 14:27:20.463275 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:20.463220    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podStartSLOduration=2.376244742 podStartE2EDuration="6.463205972s" totalImagesPullingTime="4.08696123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:27:20.461636192 +0000 UTC m=+580.627121489" watchObservedRunningTime="2026-11-01 14:27:20.463205972 +0000 UTC m=+580.628691328"
Nov 01 14:27:21.213665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:21.213636    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:21.222520 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:21.222489    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-drf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 14:27:21.222736 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:21.222713    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 14:27:22.211770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:22.211734    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-drf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 14:27:22.212057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:22.212027    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 14:27:23.214410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:23.213654    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-drf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 14:27:23.214410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:23.213719    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 14:27:24.219742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:24.219698    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-drf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 14:27:24.220120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:24.219761    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 14:27:25.214409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:25.212268    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-drf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 14:27:25.214409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:25.212322    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 14:27:26.211593 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:26.211555    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-drf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 14:27:26.212017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:26.211624    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 14:27:27.806916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:27.806864    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:27.807376 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:27.807345    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:27.808020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:27.807988    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:27:28.450947 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.450912    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d985f759b-hcptd"]
Nov 01 14:27:28.454622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.454599    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d985f759b-hcptd"
Nov 01 14:27:28.506459 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.506375    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d985f759b-hcptd"]
Nov 01 14:27:28.506996 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.506965    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9sr64\""
Nov 01 14:27:28.506996 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.506984    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 14:27:28.576551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.576528    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/590e6cf3-e96c-4b31-a570-555017c2bb47-tls-cert\") pod \"authorino-d985f759b-hcptd\" (UID: \"590e6cf3-e96c-4b31-a570-555017c2bb47\") " pod="kuadrant-system/authorino-d985f759b-hcptd"
Nov 01 14:27:28.576677 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.576568    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp7wl\" (UniqueName: \"kubernetes.io/projected/590e6cf3-e96c-4b31-a570-555017c2bb47-kube-api-access-lp7wl\") pod \"authorino-d985f759b-hcptd\" (UID: \"590e6cf3-e96c-4b31-a570-555017c2bb47\") " pod="kuadrant-system/authorino-d985f759b-hcptd"
Nov 01 14:27:28.677282 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.677254    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/590e6cf3-e96c-4b31-a570-555017c2bb47-tls-cert\") pod \"authorino-d985f759b-hcptd\" (UID: \"590e6cf3-e96c-4b31-a570-555017c2bb47\") " pod="kuadrant-system/authorino-d985f759b-hcptd"
Nov 01 14:27:28.677465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.677295    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7wl\" (UniqueName: \"kubernetes.io/projected/590e6cf3-e96c-4b31-a570-555017c2bb47-kube-api-access-lp7wl\") pod \"authorino-d985f759b-hcptd\" (UID: \"590e6cf3-e96c-4b31-a570-555017c2bb47\") " pod="kuadrant-system/authorino-d985f759b-hcptd"
Nov 01 14:27:28.679721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.679697    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/590e6cf3-e96c-4b31-a570-555017c2bb47-tls-cert\") pod \"authorino-d985f759b-hcptd\" (UID: \"590e6cf3-e96c-4b31-a570-555017c2bb47\") " pod="kuadrant-system/authorino-d985f759b-hcptd"
Nov 01 14:27:28.733732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.733667    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp7wl\" (UniqueName: \"kubernetes.io/projected/590e6cf3-e96c-4b31-a570-555017c2bb47-kube-api-access-lp7wl\") pod \"authorino-d985f759b-hcptd\" (UID: \"590e6cf3-e96c-4b31-a570-555017c2bb47\") " pod="kuadrant-system/authorino-d985f759b-hcptd"
Nov 01 14:27:28.765552 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:28.765524    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d985f759b-hcptd"
Nov 01 14:27:29.127664 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:27:29.127594    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod590e6cf3_e96c_4b31_a570_555017c2bb47.slice/crio-94e3d2687a556ee28c0932eeb8f708c841c480bf0509932f424ff53527286b59 WatchSource:0}: Error finding container 94e3d2687a556ee28c0932eeb8f708c841c480bf0509932f424ff53527286b59: Status 404 returned error can't find the container with id 94e3d2687a556ee28c0932eeb8f708c841c480bf0509932f424ff53527286b59
Nov 01 14:27:29.129194 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:29.129166    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d985f759b-hcptd"]
Nov 01 14:27:29.357457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:29.357421    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d985f759b-hcptd" event={"ID":"590e6cf3-e96c-4b31-a570-555017c2bb47","Type":"ContainerStarted","Data":"94e3d2687a556ee28c0932eeb8f708c841c480bf0509932f424ff53527286b59"}
Nov 01 14:27:30.025047 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.025017    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"]
Nov 01 14:27:30.029788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.029756    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.190813 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.190783    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.191182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.190829    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.191182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.190861    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.191182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.190938    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.191182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.191027    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.191182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.191048    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.191182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.191064    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hw82\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-kube-api-access-9hw82\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.191182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.191085    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.191463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.191192    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.227034 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.227007    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"]
Nov 01 14:27:30.292489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292425    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292459    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292499    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292606    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292637    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hw82\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-kube-api-access-9hw82\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292657    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292699    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292743    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292803    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292856    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.292886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292873    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.293143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.292998    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.293143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.293103    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.293304 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.293284    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.295097 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.295075    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.295194 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.295148    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.349419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.349372    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.349668 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.349652    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hw82\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-kube-api-access-9hw82\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rt5lp\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:30.642959 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:30.642932    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:31.068089 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:27:31.068053    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b9ce89_85c6_4df7_a6d8_a1bdbb858b24.slice/crio-1492cdd24150aebfd50a6022875a3de7db44bea46d1516121460e8ca953c706b WatchSource:0}: Error finding container 1492cdd24150aebfd50a6022875a3de7db44bea46d1516121460e8ca953c706b: Status 404 returned error can't find the container with id 1492cdd24150aebfd50a6022875a3de7db44bea46d1516121460e8ca953c706b
Nov 01 14:27:31.068684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.068661    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"]
Nov 01 14:27:31.071480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.071448    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:27:31.071570 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.071513    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:27:31.071570 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.071541    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:27:31.369507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.369423    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d985f759b-hcptd" event={"ID":"590e6cf3-e96c-4b31-a570-555017c2bb47","Type":"ContainerStarted","Data":"5fa09320b094b18c959498280252f7fe26676f17ed8551a2768e4ac887eb53d4"}
Nov 01 14:27:31.371112 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.371082    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" event={"ID":"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24","Type":"ContainerStarted","Data":"f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88"}
Nov 01 14:27:31.371240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.371135    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" event={"ID":"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24","Type":"ContainerStarted","Data":"1492cdd24150aebfd50a6022875a3de7db44bea46d1516121460e8ca953c706b"}
Nov 01 14:27:31.483343 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.483272    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d985f759b-hcptd" podStartSLOduration=1.974237527 podStartE2EDuration="3.483250733s" totalImagesPullingTime="1.509013206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:27:31.481089955 +0000 UTC m=+591.646575253" watchObservedRunningTime="2026-11-01 14:27:31.483250733 +0000 UTC m=+591.648736049"
Nov 01 14:27:31.598315 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.598249    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podStartSLOduration=2.5982317139999997 podStartE2EDuration="2.598231714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:27:31.596218456 +0000 UTC m=+591.761703755" watchObservedRunningTime="2026-11-01 14:27:31.598231714 +0000 UTC m=+591.763717013"
Nov 01 14:27:31.643469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:31.643375    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:32.644551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:32.644508    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rt5lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 14:27:32.644944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:32.644567    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 14:27:33.643982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:33.643948    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rt5lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 14:27:33.644156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:33.644002    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 14:27:34.643693 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:34.643657    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rt5lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 14:27:34.644078 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:34.643716    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 14:27:35.148914 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:27:35.148865    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:27:35.643314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:35.643268    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rt5lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 14:27:35.643531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:35.643320    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 14:27:36.644018 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:36.643980    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rt5lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 14:27:36.644442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:36.644037    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 14:27:37.643467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:37.643430    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rt5lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 14:27:37.643633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:37.643488    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 14:27:38.643894 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:38.643856    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rt5lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 14:27:38.644302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:38.643909    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 14:27:39.647960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:39.647930    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:39.648335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:39.648311    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:39.649096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:39.649078    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:27:40.018469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:40.018372    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:27:40.019149 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:40.019122    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:27:58.954601 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:58.954565    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"]
Nov 01 14:27:58.958321 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:58.958300    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.034905 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.034867    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"]
Nov 01 14:27:59.062399 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.062368    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f7c71759-openshift-default-dockercfg-gz99v\""
Nov 01 14:27:59.090162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090116    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-envoy\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.090336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090184    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-credential-socket\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.090336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090283    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-token\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.090336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090323    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5139751-71a0-4fd3-953c-99fdcd05fb44-istiod-ca-cert\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.090560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090428    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqc9\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-kube-api-access-thqc9\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.090560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090514    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-data\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.090560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090543    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-certs\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.090737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090604    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-socket\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.090737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.090637    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-podinfo\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.191828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.191793    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-socket\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.191828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.191834    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-podinfo\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.191869    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-envoy\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.191893    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-credential-socket\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.191925    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-token\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.191945    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5139751-71a0-4fd3-953c-99fdcd05fb44-istiod-ca-cert\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.191994    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqc9\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-kube-api-access-thqc9\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.192036    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-data\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.192060    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-certs\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.192480    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-socket\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.192552    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-credential-socket\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.192620    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-certs\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.192675    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-data\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.192805 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.192787    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5139751-71a0-4fd3-953c-99fdcd05fb44-istiod-ca-cert\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.194743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.194722    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-envoy\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.195062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.195039    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-podinfo\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.249509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.249442    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-token\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.252156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.252133    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqc9\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-kube-api-access-thqc9\") pod \"e2e-ait-f7c71759-openshift-default-54785b7659-62xsc\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") " pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.271887 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.271862    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:27:59.712991 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:27:59.712962    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5139751_71a0_4fd3_953c_99fdcd05fb44.slice/crio-22c8c50c145b1d49d233ce185672c3d1f0a34cd3dcda14a281a184d8a0cd349b WatchSource:0}: Error finding container 22c8c50c145b1d49d233ce185672c3d1f0a34cd3dcda14a281a184d8a0cd349b: Status 404 returned error can't find the container with id 22c8c50c145b1d49d233ce185672c3d1f0a34cd3dcda14a281a184d8a0cd349b
Nov 01 14:27:59.714248 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.714220    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"]
Nov 01 14:27:59.715189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.715151    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:27:59.715298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.715225    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:27:59.715298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:27:59.715260    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:00.534933 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:00.534890    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" event={"ID":"a5139751-71a0-4fd3-953c-99fdcd05fb44","Type":"ContainerStarted","Data":"ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00"}
Nov 01 14:28:00.535309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:00.534959    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" event={"ID":"a5139751-71a0-4fd3-953c-99fdcd05fb44","Type":"ContainerStarted","Data":"22c8c50c145b1d49d233ce185672c3d1f0a34cd3dcda14a281a184d8a0cd349b"}
Nov 01 14:28:00.651562 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:00.651509    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" podStartSLOduration=2.651493837 podStartE2EDuration="2.651493837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:00.650353231 +0000 UTC m=+620.815838529" watchObservedRunningTime="2026-11-01 14:28:00.651493837 +0000 UTC m=+620.816979226"
Nov 01 14:28:00.913439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:00.913405    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"]
Nov 01 14:28:00.917248 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:00.917228    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:00.971779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:00.971748    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"]
Nov 01 14:28:01.014566 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.014540    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-envoy\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.014722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.014589    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-certs\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.014722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.014698    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-credential-socket\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.014722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.014717    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istiod-ca-cert\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.015099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.015074    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc9h9\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-kube-api-access-rc9h9\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.015185 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.015159    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-token\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.015237 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.015218    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-socket\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.015277 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.015244    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-data\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.015320 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.015278    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-podinfo\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.018633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.018617    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d8f1c66f-openshift-default-dockercfg-qhcf4\""
Nov 01 14:28:01.116596 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116556    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-socket\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.116793 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116607    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-data\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.116793 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116649    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-podinfo\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.116793 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116677    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-envoy\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.116793 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116711    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-certs\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.116793 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116740    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-credential-socket\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.116793 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116766    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istiod-ca-cert\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.117105 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116834    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc9h9\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-kube-api-access-rc9h9\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.117105 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.116889    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-token\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.117105 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.117016    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-socket\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.117105 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.117064    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-data\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.117303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.117109    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-credential-socket\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.117468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.117447    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-certs\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.117730 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.117707    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istiod-ca-cert\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.119072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.119049    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-envoy\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.119274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.119257    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-podinfo\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.174122 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.174063    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-token\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.174343 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.174324    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc9h9\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-kube-api-access-rc9h9\") pod \"e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") " pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.230164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.230131    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:01.272305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.272272    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:28:01.277424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.277403    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:28:01.540301 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.540214    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:28:01.541088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.541072    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:28:01.645379 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.645346    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:01.645507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.645429    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:01.645507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.645459    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:01.645619 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:01.645600    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"]
Nov 01 14:28:02.546643 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:02.546606    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" event={"ID":"b4f3576f-f0a4-4eb2-a770-6b52af66d488","Type":"ContainerStarted","Data":"c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426"}
Nov 01 14:28:02.547037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:02.546684    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" event={"ID":"b4f3576f-f0a4-4eb2-a770-6b52af66d488","Type":"ContainerStarted","Data":"40e505b12afc66bc8a17d2dcc354685d109acfbd3d389ceee9fc037ade2f9018"}
Nov 01 14:28:02.667647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:02.667598    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podStartSLOduration=2.667583928 podStartE2EDuration="2.667583928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:02.666683591 +0000 UTC m=+622.832168890" watchObservedRunningTime="2026-11-01 14:28:02.667583928 +0000 UTC m=+622.833069225"
Nov 01 14:28:03.231345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:03.231317    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:03.236355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:03.236331    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:03.552286 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:03.552196    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:03.553936 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:03.553915    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:05.149322 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:28:05.149289    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:28:05.854744 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:05.854713    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"]
Nov 01 14:28:05.858447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:05.858430    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:05.957173 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:05.957014    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e27521d\""
Nov 01 14:28:05.960090 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:05.960056    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"]
Nov 01 14:28:05.961596 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:05.961576    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e27521d-dockercfg-49cpm\""
Nov 01 14:28:05.969049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:05.969023    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fbdfa1-99e3-43a1-a776-8c7a2827272d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd\" (UID: \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:05.969190 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:05.969168    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k57qp\" (UniqueName: \"kubernetes.io/projected/30fbdfa1-99e3-43a1-a776-8c7a2827272d-kube-api-access-k57qp\") pod \"payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd\" (UID: \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:06.029174 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.029148    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"]
Nov 01 14:28:06.032688 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.032673    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:06.069800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.069769    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fbdfa1-99e3-43a1-a776-8c7a2827272d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd\" (UID: \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:06.069909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.069855    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k57qp\" (UniqueName: \"kubernetes.io/projected/30fbdfa1-99e3-43a1-a776-8c7a2827272d-kube-api-access-k57qp\") pod \"payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd\" (UID: \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:06.070455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.070419    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fbdfa1-99e3-43a1-a776-8c7a2827272d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd\" (UID: \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:06.085698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.085671    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"]
Nov 01 14:28:06.127075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.127018    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k57qp\" (UniqueName: \"kubernetes.io/projected/30fbdfa1-99e3-43a1-a776-8c7a2827272d-kube-api-access-k57qp\") pod \"payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd\" (UID: \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:06.170966 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.170932    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c859657a-731c-4cdb-a563-b716fa6b41ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj\" (UID: \"c859657a-731c-4cdb-a563-b716fa6b41ae\") " pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:06.171354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.170979    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jlg9\" (UniqueName: \"kubernetes.io/projected/c859657a-731c-4cdb-a563-b716fa6b41ae-kube-api-access-9jlg9\") pod \"payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj\" (UID: \"c859657a-731c-4cdb-a563-b716fa6b41ae\") " pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:06.171354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.171066    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:06.272361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.272331    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c859657a-731c-4cdb-a563-b716fa6b41ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj\" (UID: \"c859657a-731c-4cdb-a563-b716fa6b41ae\") " pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:06.272554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.272395    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jlg9\" (UniqueName: \"kubernetes.io/projected/c859657a-731c-4cdb-a563-b716fa6b41ae-kube-api-access-9jlg9\") pod \"payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj\" (UID: \"c859657a-731c-4cdb-a563-b716fa6b41ae\") " pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:06.273232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.273209    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c859657a-731c-4cdb-a563-b716fa6b41ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj\" (UID: \"c859657a-731c-4cdb-a563-b716fa6b41ae\") " pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:06.329036 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.329003    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jlg9\" (UniqueName: \"kubernetes.io/projected/c859657a-731c-4cdb-a563-b716fa6b41ae-kube-api-access-9jlg9\") pod \"payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj\" (UID: \"c859657a-731c-4cdb-a563-b716fa6b41ae\") " pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:06.346483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.346461    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:06.537554 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:28:06.537518    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fbdfa1_99e3_43a1_a776_8c7a2827272d.slice/crio-8d2e0a84c64ae3d8c248cb22e37dab9a85aea9cef899913bcbd36c8090af5c58 WatchSource:0}: Error finding container 8d2e0a84c64ae3d8c248cb22e37dab9a85aea9cef899913bcbd36c8090af5c58: Status 404 returned error can't find the container with id 8d2e0a84c64ae3d8c248cb22e37dab9a85aea9cef899913bcbd36c8090af5c58
Nov 01 14:28:06.538745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.538723    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"]
Nov 01 14:28:06.571368 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.571344    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd" event={"ID":"30fbdfa1-99e3-43a1-a776-8c7a2827272d","Type":"ContainerStarted","Data":"8d2e0a84c64ae3d8c248cb22e37dab9a85aea9cef899913bcbd36c8090af5c58"}
Nov 01 14:28:06.711451 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:28:06.711421    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc859657a_731c_4cdb_a563_b716fa6b41ae.slice/crio-5d41d2779eb02f64da9f8a12fbe1065d7177d1b614c76921001cf08c14006c33 WatchSource:0}: Error finding container 5d41d2779eb02f64da9f8a12fbe1065d7177d1b614c76921001cf08c14006c33: Status 404 returned error can't find the container with id 5d41d2779eb02f64da9f8a12fbe1065d7177d1b614c76921001cf08c14006c33
Nov 01 14:28:06.713244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:06.713220    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"]
Nov 01 14:28:07.578072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:07.578030    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj" event={"ID":"c859657a-731c-4cdb-a563-b716fa6b41ae","Type":"ContainerStarted","Data":"0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6"}
Nov 01 14:28:07.578472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:07.578088    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:07.578472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:07.578104    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj" event={"ID":"c859657a-731c-4cdb-a563-b716fa6b41ae","Type":"ContainerStarted","Data":"5d41d2779eb02f64da9f8a12fbe1065d7177d1b614c76921001cf08c14006c33"}
Nov 01 14:28:07.579938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:07.579901    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd" event={"ID":"30fbdfa1-99e3-43a1-a776-8c7a2827272d","Type":"ContainerStarted","Data":"3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2"}
Nov 01 14:28:07.580093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:07.579953    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:07.693986 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:07.693934    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj" podStartSLOduration=2.6939195 podStartE2EDuration="2.6939195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:07.692675508 +0000 UTC m=+627.858160805" watchObservedRunningTime="2026-11-01 14:28:07.6939195 +0000 UTC m=+627.859404798"
Nov 01 14:28:10.362634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.362576    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd" podStartSLOduration=5.362561726 podStartE2EDuration="5.362561726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:07.81485908 +0000 UTC m=+627.980344383" watchObservedRunningTime="2026-11-01 14:28:10.362561726 +0000 UTC m=+630.528047024"
Nov 01 14:28:10.363958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.363937    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"]
Nov 01 14:28:10.367895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.367876    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:10.423158 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.423078    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"]
Nov 01 14:28:10.471403 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.471191    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f7c71759\""
Nov 01 14:28:10.471566 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.471419    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f7c71759-dockercfg-2wgh2\""
Nov 01 14:28:10.521530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.521494    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtf9\" (UniqueName: \"kubernetes.io/projected/43baaf8d-6864-4dd4-94e4-047862b9873e-kube-api-access-mgtf9\") pod \"payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t\" (UID: \"43baaf8d-6864-4dd4-94e4-047862b9873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:10.521708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.521578    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43baaf8d-6864-4dd4-94e4-047862b9873e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t\" (UID: \"43baaf8d-6864-4dd4-94e4-047862b9873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:10.533752 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.533720    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"]
Nov 01 14:28:10.537259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.537235    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:10.589964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.589925    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"]
Nov 01 14:28:10.623139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.623063    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6d5h\" (UniqueName: \"kubernetes.io/projected/84a57c88-db54-46a9-899c-179896978cf8-kube-api-access-g6d5h\") pod \"payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2\" (UID: \"84a57c88-db54-46a9-899c-179896978cf8\") " pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:10.623139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.623105    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtf9\" (UniqueName: \"kubernetes.io/projected/43baaf8d-6864-4dd4-94e4-047862b9873e-kube-api-access-mgtf9\") pod \"payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t\" (UID: \"43baaf8d-6864-4dd4-94e4-047862b9873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:10.623139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.623138    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a57c88-db54-46a9-899c-179896978cf8-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2\" (UID: \"84a57c88-db54-46a9-899c-179896978cf8\") " pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:10.623348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.623327    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43baaf8d-6864-4dd4-94e4-047862b9873e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t\" (UID: \"43baaf8d-6864-4dd4-94e4-047862b9873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:10.624048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.624031    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43baaf8d-6864-4dd4-94e4-047862b9873e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t\" (UID: \"43baaf8d-6864-4dd4-94e4-047862b9873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:10.679678 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.679655    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtf9\" (UniqueName: \"kubernetes.io/projected/43baaf8d-6864-4dd4-94e4-047862b9873e-kube-api-access-mgtf9\") pod \"payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t\" (UID: \"43baaf8d-6864-4dd4-94e4-047862b9873e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:10.681507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.681486    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:10.724652 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.724620    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a57c88-db54-46a9-899c-179896978cf8-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2\" (UID: \"84a57c88-db54-46a9-899c-179896978cf8\") " pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:10.724779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.724730    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6d5h\" (UniqueName: \"kubernetes.io/projected/84a57c88-db54-46a9-899c-179896978cf8-kube-api-access-g6d5h\") pod \"payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2\" (UID: \"84a57c88-db54-46a9-899c-179896978cf8\") " pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:10.725593 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.725573    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a57c88-db54-46a9-899c-179896978cf8-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2\" (UID: \"84a57c88-db54-46a9-899c-179896978cf8\") " pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:10.782374 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.781910    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6d5h\" (UniqueName: \"kubernetes.io/projected/84a57c88-db54-46a9-899c-179896978cf8-kube-api-access-g6d5h\") pod \"payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2\" (UID: \"84a57c88-db54-46a9-899c-179896978cf8\") " pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:10.850665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:10.850635    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:11.052287 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:28:11.052256    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43baaf8d_6864_4dd4_94e4_047862b9873e.slice/crio-e588f005328cee318806932bdeb159f55614f179eeb69a54beb655aa8b8a2ec1 WatchSource:0}: Error finding container e588f005328cee318806932bdeb159f55614f179eeb69a54beb655aa8b8a2ec1: Status 404 returned error can't find the container with id e588f005328cee318806932bdeb159f55614f179eeb69a54beb655aa8b8a2ec1
Nov 01 14:28:11.053949 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.053930    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"]
Nov 01 14:28:11.215430 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:28:11.215398    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a57c88_db54_46a9_899c_179896978cf8.slice/crio-99ee45d41c9834425ea88c0a64dc93632d4eacd75c2c56cec4e01235268a2b69 WatchSource:0}: Error finding container 99ee45d41c9834425ea88c0a64dc93632d4eacd75c2c56cec4e01235268a2b69: Status 404 returned error can't find the container with id 99ee45d41c9834425ea88c0a64dc93632d4eacd75c2c56cec4e01235268a2b69
Nov 01 14:28:11.217032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.217000    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"]
Nov 01 14:28:11.600257 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.600211    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2" event={"ID":"84a57c88-db54-46a9-899c-179896978cf8","Type":"ContainerStarted","Data":"469f59c73e807114430c872566e26d52d915801b6756bdac2aa93468df108c5d"}
Nov 01 14:28:11.600727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.600274    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2" event={"ID":"84a57c88-db54-46a9-899c-179896978cf8","Type":"ContainerStarted","Data":"99ee45d41c9834425ea88c0a64dc93632d4eacd75c2c56cec4e01235268a2b69"}
Nov 01 14:28:11.600727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.600376    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:11.601772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.601747    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t" event={"ID":"43baaf8d-6864-4dd4-94e4-047862b9873e","Type":"ContainerStarted","Data":"a987cb4b199d56b1e12187b7caf936b25565ecaf8245c8fa189623fc3d6c76ba"}
Nov 01 14:28:11.601876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.601793    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t" event={"ID":"43baaf8d-6864-4dd4-94e4-047862b9873e","Type":"ContainerStarted","Data":"e588f005328cee318806932bdeb159f55614f179eeb69a54beb655aa8b8a2ec1"}
Nov 01 14:28:11.601876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.601838    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:11.714030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.713982    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2" podStartSLOduration=1.713968678 podStartE2EDuration="1.713968678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:11.712695242 +0000 UTC m=+631.878180539" watchObservedRunningTime="2026-11-01 14:28:11.713968678 +0000 UTC m=+631.879453997"
Nov 01 14:28:11.832143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:11.832071    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t" podStartSLOduration=1.832048626 podStartE2EDuration="1.832048626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:11.829621832 +0000 UTC m=+631.995107131" watchObservedRunningTime="2026-11-01 14:28:11.832048626 +0000 UTC m=+631.997533927"
Nov 01 14:28:13.191278 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.191219    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"]
Nov 01 14:28:13.194862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.194846    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.255299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.255200    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d8f1c66f-serving-cert\""
Nov 01 14:28:13.257785 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.257758    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"]
Nov 01 14:28:13.350176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.350141    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c943d664-5c89-4a8e-9e56-5dbea389f201-openshift-service-ca\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.350319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.350185    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.350377 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.350311    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2l9w\" (UniqueName: \"kubernetes.io/projected/c943d664-5c89-4a8e-9e56-5dbea389f201-kube-api-access-z2l9w\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.350377 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.350364    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-metrics-certs\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.451971 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.451886    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-metrics-certs\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.451971 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.451956    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c943d664-5c89-4a8e-9e56-5dbea389f201-openshift-service-ca\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.452178 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.451990    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.452178 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.452049    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2l9w\" (UniqueName: \"kubernetes.io/projected/c943d664-5c89-4a8e-9e56-5dbea389f201-kube-api-access-z2l9w\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.452178 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:28:13.452145    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-serving-cert: secret "maas-api-e2e-ait-d8f1c66f-serving-cert" not found
Nov 01 14:28:13.452317 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:28:13.452235    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls podName:c943d664-5c89-4a8e-9e56-5dbea389f201 nodeName:}" failed. No retries permitted until 2026-11-01 14:28:13.9522101 +0000 UTC m=+634.117695392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls") pod "maas-api-e2e-ait-d8f1c66f-766887df7-t54d9" (UID: "c943d664-5c89-4a8e-9e56-5dbea389f201") : secret "maas-api-e2e-ait-d8f1c66f-serving-cert" not found
Nov 01 14:28:13.452721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.452699    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c943d664-5c89-4a8e-9e56-5dbea389f201-openshift-service-ca\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.454465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.454448    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-metrics-certs\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.509903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.509869    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2l9w\" (UniqueName: \"kubernetes.io/projected/c943d664-5c89-4a8e-9e56-5dbea389f201-kube-api-access-z2l9w\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.957220 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.957184    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:13.959627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:13.959599    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls\") pod \"maas-api-e2e-ait-d8f1c66f-766887df7-t54d9\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:14.109109 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:14.109078    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:14.479655 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:28:14.479621    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc943d664_5c89_4a8e_9e56_5dbea389f201.slice/crio-caaad28ea7d57217e66062a77d223a05150d235bcf9bcd9f6486cc7abbf51a32 WatchSource:0}: Error finding container caaad28ea7d57217e66062a77d223a05150d235bcf9bcd9f6486cc7abbf51a32: Status 404 returned error can't find the container with id caaad28ea7d57217e66062a77d223a05150d235bcf9bcd9f6486cc7abbf51a32
Nov 01 14:28:14.481022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:14.481003    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:28:14.481554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:14.481529    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"]
Nov 01 14:28:14.617508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:14.617471    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9" event={"ID":"c943d664-5c89-4a8e-9e56-5dbea389f201","Type":"ContainerStarted","Data":"caaad28ea7d57217e66062a77d223a05150d235bcf9bcd9f6486cc7abbf51a32"}
Nov 01 14:28:15.623992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:15.623952    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9" event={"ID":"c943d664-5c89-4a8e-9e56-5dbea389f201","Type":"ContainerStarted","Data":"c81aa90c0ca83d0e18ddaeef70407ba3bc5525a1fd85c76dc393b92662054bc4"}
Nov 01 14:28:15.624450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:15.624148    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:15.739831 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:15.739729    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9" podStartSLOduration=1.843785471 podStartE2EDuration="2.739713847s" totalImagesPullingTime="895.928376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:15.737982801 +0000 UTC m=+635.903468101" watchObservedRunningTime="2026-11-01 14:28:15.739713847 +0000 UTC m=+635.905199145"
Nov 01 14:28:18.587841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:18.587812    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:28:18.588221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:18.587877    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:28:18.959788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:18.959711    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:28:21.634837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:21.634764    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:28:22.620575 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.620541    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:28:22.620890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.620871    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:28:22.859753 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.859714    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"]
Nov 01 14:28:22.863517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.863491    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:22.920228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.920183    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c9fe7b2a-serving-cert\""
Nov 01 14:28:22.952059 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.952032    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"]
Nov 01 14:28:22.958918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.958880    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:22.959060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.958945    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-metrics-certs\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:22.959060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.958985    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-openshift-service-ca\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:22.959060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:22.959029    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52h4m\" (UniqueName: \"kubernetes.io/projected/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-kube-api-access-52h4m\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.059982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.059952    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-metrics-certs\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.060132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.059994    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-openshift-service-ca\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.060132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.060024    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52h4m\" (UniqueName: \"kubernetes.io/projected/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-kube-api-access-52h4m\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.060210 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.060128    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.060252 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:28:23.060227    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-serving-cert: secret "maas-api-e2e-migrate-c9fe7b2a-serving-cert" not found
Nov 01 14:28:23.060331 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:28:23.060319    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls podName:17244c10-c2e8-4fbf-85a9-0c1342e3c72a nodeName:}" failed. No retries permitted until 2026-11-01 14:28:23.560297743 +0000 UTC m=+643.725783035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls") pod "maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp" (UID: "17244c10-c2e8-4fbf-85a9-0c1342e3c72a") : secret "maas-api-e2e-migrate-c9fe7b2a-serving-cert" not found
Nov 01 14:28:23.060792 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.060777    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-openshift-service-ca\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.062253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.062234    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-metrics-certs\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.126456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.123982    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52h4m\" (UniqueName: \"kubernetes.io/projected/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-kube-api-access-52h4m\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.565194 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.565162    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.567670 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.567641    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls\") pod \"maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:23.778737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:23.778693    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:24.171838 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:28:24.171809    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17244c10_c2e8_4fbf_85a9_0c1342e3c72a.slice/crio-62fb6ce4e80c92a3c4c4cdc63c15a590095b04b9e76ab21eeb0153f794644855 WatchSource:0}: Error finding container 62fb6ce4e80c92a3c4c4cdc63c15a590095b04b9e76ab21eeb0153f794644855: Status 404 returned error can't find the container with id 62fb6ce4e80c92a3c4c4cdc63c15a590095b04b9e76ab21eeb0153f794644855
Nov 01 14:28:24.172284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:24.172261    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"]
Nov 01 14:28:24.672658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:24.672621    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp" event={"ID":"17244c10-c2e8-4fbf-85a9-0c1342e3c72a","Type":"ContainerStarted","Data":"62fb6ce4e80c92a3c4c4cdc63c15a590095b04b9e76ab21eeb0153f794644855"}
Nov 01 14:28:25.708285 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:25.708242    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp" event={"ID":"17244c10-c2e8-4fbf-85a9-0c1342e3c72a","Type":"ContainerStarted","Data":"05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682"}
Nov 01 14:28:25.708825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:25.708367    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:27.255523 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:27.255469    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp" podStartSLOduration=4.344629166 podStartE2EDuration="5.255455136s" totalImagesPullingTime="910.82597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:25.826634477 +0000 UTC m=+645.992119803" watchObservedRunningTime="2026-11-01 14:28:27.255455136 +0000 UTC m=+647.420940433"
Nov 01 14:28:27.255949 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:27.255715    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:28:28.733840 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:28.733738    5423 generic.go:356] "Generic (PLEG): container finished" podID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerID="ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00" exitCode=0
Nov 01 14:28:28.733840 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:28.733811    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" event={"ID":"a5139751-71a0-4fd3-953c-99fdcd05fb44","Type":"ContainerDied","Data":"ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00"}
Nov 01 14:28:28.734352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:28.734192    5423 scope.go:122] "RemoveContainer" containerID="ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00"
Nov 01 14:28:28.735521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:28.735499    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:28.735574 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:28.735562    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:28.735608 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:28.735593    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:29.749280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:29.749248    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" event={"ID":"a5139751-71a0-4fd3-953c-99fdcd05fb44","Type":"ContainerStarted","Data":"1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19"}
Nov 01 14:28:30.272216 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:30.272136    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:28:30.273296 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:30.273265    5423 patch_prober.go:29] interesting pod/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 14:28:30.273433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:30.273331    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 14:28:31.272308 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:31.272271    5423 patch_prober.go:29] interesting pod/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 14:28:31.272729 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:31.272323    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 14:28:31.540627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:31.540532    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:28:31.735628 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:31.735593    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:28:32.272325 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:32.272294    5423 patch_prober.go:29] interesting pod/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 14:28:32.272749 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:32.272343    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 14:28:33.272709 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:33.272669    5423 patch_prober.go:29] interesting pod/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 14:28:33.273166 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:33.272737    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 14:28:34.272451 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:34.272414    5423 patch_prober.go:29] interesting pod/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 01 14:28:34.272649 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:34.272478    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 01 14:28:34.616409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:34.614521    5423 patch_prober.go:29] interesting pod/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:28:34.616409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:34.614622    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:28:35.149152 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:28:35.149110    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:28:35.277172 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:35.277141    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:28:35.278151 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:35.278130    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:28:36.656287 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:36.656255    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:28:36.788083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:36.787996    5423 generic.go:356] "Generic (PLEG): container finished" podID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerID="c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426" exitCode=0
Nov 01 14:28:36.788254 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:36.788076    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" event={"ID":"b4f3576f-f0a4-4eb2-a770-6b52af66d488","Type":"ContainerDied","Data":"c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426"}
Nov 01 14:28:36.788487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:36.788465    5423 scope.go:122] "RemoveContainer" containerID="c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426"
Nov 01 14:28:36.789734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:36.789710    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:36.789804 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:36.789781    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:36.789843 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:36.789813    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:28:37.807410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:37.806764    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" event={"ID":"b4f3576f-f0a4-4eb2-a770-6b52af66d488","Type":"ContainerStarted","Data":"fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841"}
Nov 01 14:28:38.230357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:38.230267    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:38.230531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:38.230431    5423 patch_prober.go:29] interesting pod/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 14:28:38.230531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:38.230479    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 14:28:39.231146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:39.231111    5423 patch_prober.go:29] interesting pod/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 14:28:39.231552 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:39.231171    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 14:28:40.230515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:40.230476    5423 patch_prober.go:29] interesting pod/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 14:28:40.230712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:40.230534    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 14:28:41.231136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:41.231104    5423 patch_prober.go:29] interesting pod/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 14:28:41.231678 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:41.231155    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 14:28:42.231121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:42.231082    5423 patch_prober.go:29] interesting pod/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 01 14:28:42.231296 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:42.231143    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 01 14:28:43.234037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:43.234002    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:43.234532 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:43.234297    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:28:43.234905 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:28:43.234889    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:29:05.148761 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:05.148725    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:29:05.454235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:05.454162    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"]
Nov 01 14:29:05.558762 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:05.558732    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"]
Nov 01 14:29:07.732786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:07.732749    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"]
Nov 01 14:29:07.837259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:07.837226    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"]
Nov 01 14:29:08.778060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:08.778025    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"]
Nov 01 14:29:08.778477 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:08.778237    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9" podUID="c943d664-5c89-4a8e-9e56-5dbea389f201" containerName="maas-api" containerID="cri-o://c81aa90c0ca83d0e18ddaeef70407ba3bc5525a1fd85c76dc393b92662054bc4" gracePeriod=30
Nov 01 14:29:08.981460 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:08.981430    5423 generic.go:356] "Generic (PLEG): container finished" podID="c943d664-5c89-4a8e-9e56-5dbea389f201" containerID="c81aa90c0ca83d0e18ddaeef70407ba3bc5525a1fd85c76dc393b92662054bc4" exitCode=0
Nov 01 14:29:08.981640 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:08.981512    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9" event={"ID":"c943d664-5c89-4a8e-9e56-5dbea389f201","Type":"ContainerDied","Data":"c81aa90c0ca83d0e18ddaeef70407ba3bc5525a1fd85c76dc393b92662054bc4"}
Nov 01 14:29:09.030533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.030482    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:29:09.126225 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.126194    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-metrics-certs\") pod \"c943d664-5c89-4a8e-9e56-5dbea389f201\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") "
Nov 01 14:29:09.126413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.126248    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls\") pod \"c943d664-5c89-4a8e-9e56-5dbea389f201\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") "
Nov 01 14:29:09.126413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.126271    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c943d664-5c89-4a8e-9e56-5dbea389f201-kube-api-access-z2l9w\" (UniqueName: \"kubernetes.io/projected/c943d664-5c89-4a8e-9e56-5dbea389f201-kube-api-access-z2l9w\") pod \"c943d664-5c89-4a8e-9e56-5dbea389f201\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") "
Nov 01 14:29:09.126413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.126315    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c943d664-5c89-4a8e-9e56-5dbea389f201-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c943d664-5c89-4a8e-9e56-5dbea389f201-openshift-service-ca\") pod \"c943d664-5c89-4a8e-9e56-5dbea389f201\" (UID: \"c943d664-5c89-4a8e-9e56-5dbea389f201\") "
Nov 01 14:29:09.126765 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.126737    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c943d664-5c89-4a8e-9e56-5dbea389f201-openshift-service-ca" pod "c943d664-5c89-4a8e-9e56-5dbea389f201" (UID: "c943d664-5c89-4a8e-9e56-5dbea389f201"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:09.135422 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.135360    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-metrics-certs" pod "c943d664-5c89-4a8e-9e56-5dbea389f201" (UID: "c943d664-5c89-4a8e-9e56-5dbea389f201"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:09.135646 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.135628    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls" pod "c943d664-5c89-4a8e-9e56-5dbea389f201" (UID: "c943d664-5c89-4a8e-9e56-5dbea389f201"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:09.135710 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.135690    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c943d664-5c89-4a8e-9e56-5dbea389f201-kube-api-access-z2l9w" pod "c943d664-5c89-4a8e-9e56-5dbea389f201" (UID: "c943d664-5c89-4a8e-9e56-5dbea389f201"). InnerVolumeSpecName "kube-api-access-z2l9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:09.227721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.227691    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:09.227721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.227717    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c943d664-5c89-4a8e-9e56-5dbea389f201-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:09.227918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.227728    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2l9w\" (UniqueName: \"kubernetes.io/projected/c943d664-5c89-4a8e-9e56-5dbea389f201-kube-api-access-z2l9w\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:09.227918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.227739    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c943d664-5c89-4a8e-9e56-5dbea389f201-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:09.988082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.988056    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"
Nov 01 14:29:09.988510 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.988056    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9" event={"ID":"c943d664-5c89-4a8e-9e56-5dbea389f201","Type":"ContainerDied","Data":"caaad28ea7d57217e66062a77d223a05150d235bcf9bcd9f6486cc7abbf51a32"}
Nov 01 14:29:09.988510 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:09.988191    5423 scope.go:122] "RemoveContainer" containerID="c81aa90c0ca83d0e18ddaeef70407ba3bc5525a1fd85c76dc393b92662054bc4"
Nov 01 14:29:10.153214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.153181    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"]
Nov 01 14:29:10.203183 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.203154    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d8f1c66f-766887df7-t54d9"]
Nov 01 14:29:10.455405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.455346    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj" podUID="c859657a-731c-4cdb-a563-b716fa6b41ae" containerName="payload-processing" containerID="cri-o://0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6" gracePeriod=25
Nov 01 14:29:10.560053 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.559981    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd" podUID="30fbdfa1-99e3-43a1-a776-8c7a2827272d" containerName="payload-pre-processing" containerID="cri-o://3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2" gracePeriod=25
Nov 01 14:29:10.704671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.704647    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:29:10.819438 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.819310    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:29:10.843400 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.843361    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c859657a-731c-4cdb-a563-b716fa6b41ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c859657a-731c-4cdb-a563-b716fa6b41ae-plugins-config-volume\") pod \"c859657a-731c-4cdb-a563-b716fa6b41ae\" (UID: \"c859657a-731c-4cdb-a563-b716fa6b41ae\") "
Nov 01 14:29:10.843577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.843463    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c859657a-731c-4cdb-a563-b716fa6b41ae-kube-api-access-9jlg9\" (UniqueName: \"kubernetes.io/projected/c859657a-731c-4cdb-a563-b716fa6b41ae-kube-api-access-9jlg9\") pod \"c859657a-731c-4cdb-a563-b716fa6b41ae\" (UID: \"c859657a-731c-4cdb-a563-b716fa6b41ae\") "
Nov 01 14:29:10.843757 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.843735    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c859657a-731c-4cdb-a563-b716fa6b41ae-plugins-config-volume" pod "c859657a-731c-4cdb-a563-b716fa6b41ae" (UID: "c859657a-731c-4cdb-a563-b716fa6b41ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:10.845661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.845626    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c859657a-731c-4cdb-a563-b716fa6b41ae-kube-api-access-9jlg9" pod "c859657a-731c-4cdb-a563-b716fa6b41ae" (UID: "c859657a-731c-4cdb-a563-b716fa6b41ae"). InnerVolumeSpecName "kube-api-access-9jlg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:10.944291 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.944262    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30fbdfa1-99e3-43a1-a776-8c7a2827272d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fbdfa1-99e3-43a1-a776-8c7a2827272d-plugins-config-volume\") pod \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\" (UID: \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\") "
Nov 01 14:29:10.944490 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.944370    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fbdfa1-99e3-43a1-a776-8c7a2827272d-kube-api-access-k57qp\" (UniqueName: \"kubernetes.io/projected/30fbdfa1-99e3-43a1-a776-8c7a2827272d-kube-api-access-k57qp\") pod \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\" (UID: \"30fbdfa1-99e3-43a1-a776-8c7a2827272d\") "
Nov 01 14:29:10.944711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.944692    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c859657a-731c-4cdb-a563-b716fa6b41ae-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:10.944771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.944716    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jlg9\" (UniqueName: \"kubernetes.io/projected/c859657a-731c-4cdb-a563-b716fa6b41ae-kube-api-access-9jlg9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:10.944771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.944736    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30fbdfa1-99e3-43a1-a776-8c7a2827272d-plugins-config-volume" pod "30fbdfa1-99e3-43a1-a776-8c7a2827272d" (UID: "30fbdfa1-99e3-43a1-a776-8c7a2827272d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:10.946439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.946418    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fbdfa1-99e3-43a1-a776-8c7a2827272d-kube-api-access-k57qp" pod "30fbdfa1-99e3-43a1-a776-8c7a2827272d" (UID: "30fbdfa1-99e3-43a1-a776-8c7a2827272d"). InnerVolumeSpecName "kube-api-access-k57qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:10.994455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.994363    5423 generic.go:356] "Generic (PLEG): container finished" podID="c859657a-731c-4cdb-a563-b716fa6b41ae" containerID="0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6" exitCode=0
Nov 01 14:29:10.994455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.994421    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"
Nov 01 14:29:10.994455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.994427    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj" event={"ID":"c859657a-731c-4cdb-a563-b716fa6b41ae","Type":"ContainerDied","Data":"0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6"}
Nov 01 14:29:10.994999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.994479    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj" event={"ID":"c859657a-731c-4cdb-a563-b716fa6b41ae","Type":"ContainerDied","Data":"5d41d2779eb02f64da9f8a12fbe1065d7177d1b614c76921001cf08c14006c33"}
Nov 01 14:29:10.994999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.994502    5423 scope.go:122] "RemoveContainer" containerID="0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6"
Nov 01 14:29:10.995822 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.995787    5423 generic.go:356] "Generic (PLEG): container finished" podID="30fbdfa1-99e3-43a1-a776-8c7a2827272d" containerID="3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2" exitCode=0
Nov 01 14:29:10.995896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.995835    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"
Nov 01 14:29:10.995896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.995843    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd" event={"ID":"30fbdfa1-99e3-43a1-a776-8c7a2827272d","Type":"ContainerDied","Data":"3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2"}
Nov 01 14:29:10.995896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:10.995885    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd" event={"ID":"30fbdfa1-99e3-43a1-a776-8c7a2827272d","Type":"ContainerDied","Data":"8d2e0a84c64ae3d8c248cb22e37dab9a85aea9cef899913bcbd36c8090af5c58"}
Nov 01 14:29:11.004030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.004015    5423 scope.go:122] "RemoveContainer" containerID="0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6"
Nov 01 14:29:11.004243 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:11.004218    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6\": container with ID starting with 0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6 not found: ID does not exist" containerID="0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6"
Nov 01 14:29:11.004302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.004255    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6"} err="failed to get container status \"0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6\": rpc error: code = NotFound desc = could not find container \"0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6\": container with ID starting with 0d5760976e398cee4f597a03e759694d6f83e96fb6987b82884809f3d28463d6 not found: ID does not exist"
Nov 01 14:29:11.004302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.004276    5423 scope.go:122] "RemoveContainer" containerID="3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2"
Nov 01 14:29:11.012635 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.012620    5423 scope.go:122] "RemoveContainer" containerID="3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2"
Nov 01 14:29:11.012861 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:11.012843    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2\": container with ID starting with 3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2 not found: ID does not exist" containerID="3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2"
Nov 01 14:29:11.012902 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.012868    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2"} err="failed to get container status \"3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2\": rpc error: code = NotFound desc = could not find container \"3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2\": container with ID starting with 3cb4c086fd1abfbc6b3aecf2fc129e7d75100b538574c87de383cc6e840ad8a2 not found: ID does not exist"
Nov 01 14:29:11.046312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.046271    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30fbdfa1-99e3-43a1-a776-8c7a2827272d-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:11.046312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.046303    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k57qp\" (UniqueName: \"kubernetes.io/projected/30fbdfa1-99e3-43a1-a776-8c7a2827272d-kube-api-access-k57qp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:11.048836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.048816    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"]
Nov 01 14:29:11.049039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.049001    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp" podUID="17244c10-c2e8-4fbf-85a9-0c1342e3c72a" containerName="maas-api" containerID="cri-o://05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682" gracePeriod=30
Nov 01 14:29:11.160146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.160116    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"]
Nov 01 14:29:11.164415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.164378    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e27521d-6f8dfb785f-z8rdj"]
Nov 01 14:29:11.278475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.275919    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"]
Nov 01 14:29:11.278475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.278143    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e27521d-7c9cfc9447-gdjpd"]
Nov 01 14:29:11.313875 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.313849    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:29:11.396194 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.396156    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"]
Nov 01 14:29:11.396465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.396436    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy" containerID="cri-o://fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841" gracePeriod=30
Nov 01 14:29:11.460538 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.460507    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls\") pod \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") "
Nov 01 14:29:11.460732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.460551    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-openshift-service-ca\") pod \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") "
Nov 01 14:29:11.460732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.460602    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-metrics-certs\") pod \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") "
Nov 01 14:29:11.460732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.460693    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-kube-api-access-52h4m\" (UniqueName: \"kubernetes.io/projected/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-kube-api-access-52h4m\") pod \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\" (UID: \"17244c10-c2e8-4fbf-85a9-0c1342e3c72a\") "
Nov 01 14:29:11.461017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.460989    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-openshift-service-ca" pod "17244c10-c2e8-4fbf-85a9-0c1342e3c72a" (UID: "17244c10-c2e8-4fbf-85a9-0c1342e3c72a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:11.462800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.462774    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-metrics-certs" pod "17244c10-c2e8-4fbf-85a9-0c1342e3c72a" (UID: "17244c10-c2e8-4fbf-85a9-0c1342e3c72a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:11.462800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.462796    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls" pod "17244c10-c2e8-4fbf-85a9-0c1342e3c72a" (UID: "17244c10-c2e8-4fbf-85a9-0c1342e3c72a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:11.462974 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.462774    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-kube-api-access-52h4m" pod "17244c10-c2e8-4fbf-85a9-0c1342e3c72a" (UID: "17244c10-c2e8-4fbf-85a9-0c1342e3c72a"). InnerVolumeSpecName "kube-api-access-52h4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:11.562573 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.562547    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:11.562573 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.562573    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:11.562854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.562582    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:11.562854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.562591    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52h4m\" (UniqueName: \"kubernetes.io/projected/17244c10-c2e8-4fbf-85a9-0c1342e3c72a-kube-api-access-52h4m\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:11.959933 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.959858    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30fbdfa1-99e3-43a1-a776-8c7a2827272d" path="/var/lib/kubelet/pods/30fbdfa1-99e3-43a1-a776-8c7a2827272d/volumes"
Nov 01 14:29:11.960241 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.960228    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c859657a-731c-4cdb-a563-b716fa6b41ae" path="/var/lib/kubelet/pods/c859657a-731c-4cdb-a563-b716fa6b41ae/volumes"
Nov 01 14:29:11.960666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:11.960651    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c943d664-5c89-4a8e-9e56-5dbea389f201" path="/var/lib/kubelet/pods/c943d664-5c89-4a8e-9e56-5dbea389f201/volumes"
Nov 01 14:29:12.008518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.008484    5423 generic.go:356] "Generic (PLEG): container finished" podID="17244c10-c2e8-4fbf-85a9-0c1342e3c72a" containerID="05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682" exitCode=0
Nov 01 14:29:12.008929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.008543    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"
Nov 01 14:29:12.008929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.008562    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp" event={"ID":"17244c10-c2e8-4fbf-85a9-0c1342e3c72a","Type":"ContainerDied","Data":"05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682"}
Nov 01 14:29:12.008929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.008631    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp" event={"ID":"17244c10-c2e8-4fbf-85a9-0c1342e3c72a","Type":"ContainerDied","Data":"62fb6ce4e80c92a3c4c4cdc63c15a590095b04b9e76ab21eeb0153f794644855"}
Nov 01 14:29:12.008929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.008644    5423 scope.go:122] "RemoveContainer" containerID="05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682"
Nov 01 14:29:12.017966 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.017928    5423 scope.go:122] "RemoveContainer" containerID="05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682"
Nov 01 14:29:12.018276 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:12.018254    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682\": container with ID starting with 05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682 not found: ID does not exist" containerID="05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682"
Nov 01 14:29:12.018375 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.018288    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682"} err="failed to get container status \"05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682\": rpc error: code = NotFound desc = could not find container \"05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682\": container with ID starting with 05bf98d9de63c8364976674368a4b8128f24dd30edb2634a01e41096bd5b4682 not found: ID does not exist"
Nov 01 14:29:12.180447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.177275    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"]
Nov 01 14:29:12.180447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.178950    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c9fe7b2a-54fd68dd59-5dbdp"]
Nov 01 14:29:12.695046 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695015    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"]
Nov 01 14:29:12.695483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695468    5423 cpu_manager.go:414] "containerMap: removing container" podUID="c943d664-5c89-4a8e-9e56-5dbea389f201" containerName="maas-api"
Nov 01 14:29:12.695525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695486    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c943d664-5c89-4a8e-9e56-5dbea389f201" containerName="maas-api"
Nov 01 14:29:12.695525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695497    5423 cpu_manager.go:414] "containerMap: removing container" podUID="17244c10-c2e8-4fbf-85a9-0c1342e3c72a" containerName="maas-api"
Nov 01 14:29:12.695525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695502    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17244c10-c2e8-4fbf-85a9-0c1342e3c72a" containerName="maas-api"
Nov 01 14:29:12.695525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695517    5423 cpu_manager.go:414] "containerMap: removing container" podUID="c859657a-731c-4cdb-a563-b716fa6b41ae" containerName="payload-processing"
Nov 01 14:29:12.695525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695523    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c859657a-731c-4cdb-a563-b716fa6b41ae" containerName="payload-processing"
Nov 01 14:29:12.695672 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695531    5423 cpu_manager.go:414] "containerMap: removing container" podUID="30fbdfa1-99e3-43a1-a776-8c7a2827272d" containerName="payload-pre-processing"
Nov 01 14:29:12.695672 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695536    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fbdfa1-99e3-43a1-a776-8c7a2827272d" containerName="payload-pre-processing"
Nov 01 14:29:12.695672 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695609    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="c943d664-5c89-4a8e-9e56-5dbea389f201" containerName="maas-api"
Nov 01 14:29:12.695672 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695618    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="17244c10-c2e8-4fbf-85a9-0c1342e3c72a" containerName="maas-api"
Nov 01 14:29:12.695672 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695629    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="c859657a-731c-4cdb-a563-b716fa6b41ae" containerName="payload-processing"
Nov 01 14:29:12.695672 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.695637    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fbdfa1-99e3-43a1-a776-8c7a2827272d" containerName="payload-pre-processing"
Nov 01 14:29:12.700527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.700508    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.759765 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.759413    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"]
Nov 01 14:29:12.764535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.764490    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2" podUID="84a57c88-db54-46a9-899c-179896978cf8" containerName="payload-processing" containerID="cri-o://469f59c73e807114430c872566e26d52d915801b6756bdac2aa93468df108c5d" gracePeriod=25
Nov 01 14:29:12.775745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.775685    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f8e2c-openshift-default-dockercfg-cq5wp\""
Nov 01 14:29:12.838554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.838500    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t" podUID="43baaf8d-6864-4dd4-94e4-047862b9873e" containerName="payload-pre-processing" containerID="cri-o://a987cb4b199d56b1e12187b7caf936b25565ecaf8245c8fa189623fc3d6c76ba" gracePeriod=25
Nov 01 14:29:12.875452 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875428    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.875568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875460    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.875649 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875559    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.875649 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875595    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.875649 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875617    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghl9\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-kube-api-access-sghl9\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.875761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875744    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.875797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875776    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dda68f1-989b-4643-9013-0e2962c9b4d9-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.875831 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875799    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.875864 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.875833    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.976632    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.976691    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dda68f1-989b-4643-9013-0e2962c9b4d9-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.976727    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.976756    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.976819    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.976852    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.976913    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.977177    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.977235    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghl9\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-kube-api-access-sghl9\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.977426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.977289    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.978081 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.977614    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.978467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.978218    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.978584 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.978524    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.978918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.978896    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dda68f1-989b-4643-9013-0e2962c9b4d9-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.980591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.980557    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:12.980976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:12.980957    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:13.039035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.038997    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghl9\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-kube-api-access-sghl9\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:13.040187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.040161    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:13.045266 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.045238    5423 generic.go:356] "Generic (PLEG): container finished" podID="84a57c88-db54-46a9-899c-179896978cf8" containerID="469f59c73e807114430c872566e26d52d915801b6756bdac2aa93468df108c5d" exitCode=0
Nov 01 14:29:13.045372 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.045313    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2" event={"ID":"84a57c88-db54-46a9-899c-179896978cf8","Type":"ContainerDied","Data":"469f59c73e807114430c872566e26d52d915801b6756bdac2aa93468df108c5d"}
Nov 01 14:29:13.048298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.048274    5423 generic.go:356] "Generic (PLEG): container finished" podID="43baaf8d-6864-4dd4-94e4-047862b9873e" containerID="a987cb4b199d56b1e12187b7caf936b25565ecaf8245c8fa189623fc3d6c76ba" exitCode=0
Nov 01 14:29:13.048425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.048318    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t" event={"ID":"43baaf8d-6864-4dd4-94e4-047862b9873e","Type":"ContainerDied","Data":"a987cb4b199d56b1e12187b7caf936b25565ecaf8245c8fa189623fc3d6c76ba"}
Nov 01 14:29:13.111955 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.111931    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:29:13.143624 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.143601    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:29:13.238962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.236435    5423 patch_prober.go:29] interesting pod/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 01 14:29:13.238962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.236502    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:29:13.280086 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.280044    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43baaf8d-6864-4dd4-94e4-047862b9873e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43baaf8d-6864-4dd4-94e4-047862b9873e-plugins-config-volume\") pod \"43baaf8d-6864-4dd4-94e4-047862b9873e\" (UID: \"43baaf8d-6864-4dd4-94e4-047862b9873e\") "
Nov 01 14:29:13.280086 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.280087    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a57c88-db54-46a9-899c-179896978cf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a57c88-db54-46a9-899c-179896978cf8-plugins-config-volume\") pod \"84a57c88-db54-46a9-899c-179896978cf8\" (UID: \"84a57c88-db54-46a9-899c-179896978cf8\") "
Nov 01 14:29:13.280303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.280244    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43baaf8d-6864-4dd4-94e4-047862b9873e-kube-api-access-mgtf9\" (UniqueName: \"kubernetes.io/projected/43baaf8d-6864-4dd4-94e4-047862b9873e-kube-api-access-mgtf9\") pod \"43baaf8d-6864-4dd4-94e4-047862b9873e\" (UID: \"43baaf8d-6864-4dd4-94e4-047862b9873e\") "
Nov 01 14:29:13.280346 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.280303    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a57c88-db54-46a9-899c-179896978cf8-kube-api-access-g6d5h\" (UniqueName: \"kubernetes.io/projected/84a57c88-db54-46a9-899c-179896978cf8-kube-api-access-g6d5h\") pod \"84a57c88-db54-46a9-899c-179896978cf8\" (UID: \"84a57c88-db54-46a9-899c-179896978cf8\") "
Nov 01 14:29:13.280536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.280515    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43baaf8d-6864-4dd4-94e4-047862b9873e-plugins-config-volume" pod "43baaf8d-6864-4dd4-94e4-047862b9873e" (UID: "43baaf8d-6864-4dd4-94e4-047862b9873e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:13.280596 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.280569    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a57c88-db54-46a9-899c-179896978cf8-plugins-config-volume" pod "84a57c88-db54-46a9-899c-179896978cf8" (UID: "84a57c88-db54-46a9-899c-179896978cf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:13.280676 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.280657    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43baaf8d-6864-4dd4-94e4-047862b9873e-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:13.280676 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.280676    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a57c88-db54-46a9-899c-179896978cf8-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:13.282471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.282448    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a57c88-db54-46a9-899c-179896978cf8-kube-api-access-g6d5h" pod "84a57c88-db54-46a9-899c-179896978cf8" (UID: "84a57c88-db54-46a9-899c-179896978cf8"). InnerVolumeSpecName "kube-api-access-g6d5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:13.282626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.282611    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43baaf8d-6864-4dd4-94e4-047862b9873e-kube-api-access-mgtf9" pod "43baaf8d-6864-4dd4-94e4-047862b9873e" (UID: "43baaf8d-6864-4dd4-94e4-047862b9873e"). InnerVolumeSpecName "kube-api-access-mgtf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:13.313160 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.313132    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:13.382781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.382091    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6d5h\" (UniqueName: \"kubernetes.io/projected/84a57c88-db54-46a9-899c-179896978cf8-kube-api-access-g6d5h\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:13.382781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.382127    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgtf9\" (UniqueName: \"kubernetes.io/projected/43baaf8d-6864-4dd4-94e4-047862b9873e-kube-api-access-mgtf9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:13.506666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.506591    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"]
Nov 01 14:29:13.507009 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.506996    5423 cpu_manager.go:414] "containerMap: removing container" podUID="84a57c88-db54-46a9-899c-179896978cf8" containerName="payload-processing"
Nov 01 14:29:13.507057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.507011    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a57c88-db54-46a9-899c-179896978cf8" containerName="payload-processing"
Nov 01 14:29:13.507057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.507042    5423 cpu_manager.go:414] "containerMap: removing container" podUID="43baaf8d-6864-4dd4-94e4-047862b9873e" containerName="payload-pre-processing"
Nov 01 14:29:13.507057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.507048    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43baaf8d-6864-4dd4-94e4-047862b9873e" containerName="payload-pre-processing"
Nov 01 14:29:13.507148 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.507103    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a57c88-db54-46a9-899c-179896978cf8" containerName="payload-processing"
Nov 01 14:29:13.507148 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.507115    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="43baaf8d-6864-4dd4-94e4-047862b9873e" containerName="payload-pre-processing"
Nov 01 14:29:13.510638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.510618    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.564710 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.564683    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-007270-openshift-default-dockercfg-78vkt\""
Nov 01 14:29:13.570076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.570046    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"]
Nov 01 14:29:13.684987 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.684952    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.684987 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.684995    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.685244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.685037    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.685244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.685128    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.685244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.685165    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.685357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.685260    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.685357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.685286    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.685357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.685310    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vnt7\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-kube-api-access-8vnt7\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.685512 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.685409    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786602 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786364    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786602 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786431    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786602 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786468    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vnt7\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-kube-api-access-8vnt7\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786602 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786507    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786602 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786592    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786624    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786663    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786704    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786744    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.786754    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.786944 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:29:13.786844    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dda68f1_989b_4643_9013_0e2962c9b4d9.slice/crio-3fc763e921274cea9c2b15975198e52fb5ca53022c9620b23f4d33ef3eae3506 WatchSource:0}: Error finding container 3fc763e921274cea9c2b15975198e52fb5ca53022c9620b23f4d33ef3eae3506: Status 404 returned error can't find the container with id 3fc763e921274cea9c2b15975198e52fb5ca53022c9620b23f4d33ef3eae3506
Nov 01 14:29:13.787253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.787115    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.787253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.787201    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.787654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.787610    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.787654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.787623    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.788843 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.788826    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"]
Nov 01 14:29:13.789140 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.789120    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.792369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.789526    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:13.792369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.789601    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:13.792369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.789642    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:13.792369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.789728    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.843845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.843818    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vnt7\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-kube-api-access-8vnt7\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.844265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.844242    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:13.959298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:13.959261    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17244c10-c2e8-4fbf-85a9-0c1342e3c72a" path="/var/lib/kubelet/pods/17244c10-c2e8-4fbf-85a9-0c1342e3c72a/volumes"
Nov 01 14:29:14.062991 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.062941    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" event={"ID":"7dda68f1-989b-4643-9013-0e2962c9b4d9","Type":"ContainerStarted","Data":"e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20"}
Nov 01 14:29:14.063409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.063031    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" event={"ID":"7dda68f1-989b-4643-9013-0e2962c9b4d9","Type":"ContainerStarted","Data":"3fc763e921274cea9c2b15975198e52fb5ca53022c9620b23f4d33ef3eae3506"}
Nov 01 14:29:14.065217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.065193    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"
Nov 01 14:29:14.065317 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.065212    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2" event={"ID":"84a57c88-db54-46a9-899c-179896978cf8","Type":"ContainerDied","Data":"99ee45d41c9834425ea88c0a64dc93632d4eacd75c2c56cec4e01235268a2b69"}
Nov 01 14:29:14.065317 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.065255    5423 scope.go:122] "RemoveContainer" containerID="469f59c73e807114430c872566e26d52d915801b6756bdac2aa93468df108c5d"
Nov 01 14:29:14.067153 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.067129    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t" event={"ID":"43baaf8d-6864-4dd4-94e4-047862b9873e","Type":"ContainerDied","Data":"e588f005328cee318806932bdeb159f55614f179eeb69a54beb655aa8b8a2ec1"}
Nov 01 14:29:14.067260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.067231    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"
Nov 01 14:29:14.074088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.074068    5423 scope.go:122] "RemoveContainer" containerID="a987cb4b199d56b1e12187b7caf936b25565ecaf8245c8fa189623fc3d6c76ba"
Nov 01 14:29:14.123924 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.123897    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:14.184864 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.184814    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" podStartSLOduration=2.18479979 podStartE2EDuration="2.18479979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:14.182580752 +0000 UTC m=+694.348066052" watchObservedRunningTime="2026-11-01 14:29:14.18479979 +0000 UTC m=+694.350285153"
Nov 01 14:29:14.307010 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.305695    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"]
Nov 01 14:29:14.308068 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.308040    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7c71759-6d4b4966b5-bzsg2"]
Nov 01 14:29:14.313753 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.313676    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:14.318784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.318762    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:14.415207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.415173    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"]
Nov 01 14:29:14.419785 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.419763    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7c71759-97bc8d47f-wlj5t"]
Nov 01 14:29:14.526299 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:29:14.526254    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f5dd4e_c04a_47ec_8291_5a6565e93bde.slice/crio-37528bcbdd3456a7c62874c2d4930f261c746ff278b11065c11a424cd2d8531d WatchSource:0}: Error finding container 37528bcbdd3456a7c62874c2d4930f261c746ff278b11065c11a424cd2d8531d: Status 404 returned error can't find the container with id 37528bcbdd3456a7c62874c2d4930f261c746ff278b11065c11a424cd2d8531d
Nov 01 14:29:14.528102 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.528082    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"]
Nov 01 14:29:14.528647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.528617    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:14.528702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.528678    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:14.528733 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:14.528711    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:15.075107 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.075071    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" event={"ID":"61f5dd4e-c04a-47ec-8291-5a6565e93bde","Type":"ContainerStarted","Data":"ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57"}
Nov 01 14:29:15.075622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.075124    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" event={"ID":"61f5dd4e-c04a-47ec-8291-5a6565e93bde","Type":"ContainerStarted","Data":"37528bcbdd3456a7c62874c2d4930f261c746ff278b11065c11a424cd2d8531d"}
Nov 01 14:29:15.075622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.075375    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:15.076240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.076220    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:15.124230 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.124206    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:15.129487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.129463    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:15.193661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.193610    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" podStartSLOduration=2.19359615 podStartE2EDuration="2.19359615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:15.19225677 +0000 UTC m=+695.357742068" watchObservedRunningTime="2026-11-01 14:29:15.19359615 +0000 UTC m=+695.359081520"
Nov 01 14:29:15.386914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.386834    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:29:15.959800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.959767    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43baaf8d-6864-4dd4-94e4-047862b9873e" path="/var/lib/kubelet/pods/43baaf8d-6864-4dd4-94e4-047862b9873e/volumes"
Nov 01 14:29:15.960265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:15.960244    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a57c88-db54-46a9-899c-179896978cf8" path="/var/lib/kubelet/pods/84a57c88-db54-46a9-899c-179896978cf8/volumes"
Nov 01 14:29:16.080119 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.080091    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:16.081132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.081107    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:16.643637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.643612    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:29:16.827670 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827640    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-kube-api-access-rc9h9\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-kube-api-access-rc9h9\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.827858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827696    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-credential-socket\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.827858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827759    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-socket\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.827858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827788    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-podinfo\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.827858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827821    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-envoy\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.827858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827845    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-token\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-token\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.828128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827892    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-certs\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.828128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827929    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-data\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.828128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.827992    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istiod-ca-cert\") pod \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\" (UID: \"b4f3576f-f0a4-4eb2-a770-6b52af66d488\") "
Nov 01 14:29:16.828128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.828009    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-credential-socket" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:16.828327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.828226    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-certs" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:16.828327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.828257    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-data" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:16.828449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.828400    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:16.828449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.828423    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:16.828449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.828437    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:16.828611 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.828544    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istiod-ca-cert" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:16.828702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.828684    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-socket" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:16.829847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.829820    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-kube-api-access-rc9h9" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "kube-api-access-rc9h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:16.830079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.830064    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-envoy" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:16.830297 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.830275    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-podinfo" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:29:16.830357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.830289    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-token" pod "b4f3576f-f0a4-4eb2-a770-6b52af66d488" (UID: "b4f3576f-f0a4-4eb2-a770-6b52af66d488"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:16.929707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.929678    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:16.929707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.929702    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:16.929919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.929715    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:16.929919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.929729    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:16.929919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.929741    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f3576f-f0a4-4eb2-a770-6b52af66d488-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:16.929919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:16.929753    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc9h9\" (UniqueName: \"kubernetes.io/projected/b4f3576f-f0a4-4eb2-a770-6b52af66d488-kube-api-access-rc9h9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:17.098057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.097973    5423 generic.go:356] "Generic (PLEG): container finished" podID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerID="fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841" exitCode=0
Nov 01 14:29:17.098758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.098741    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"
Nov 01 14:29:17.100003 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.099975    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" event={"ID":"b4f3576f-f0a4-4eb2-a770-6b52af66d488","Type":"ContainerDied","Data":"fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841"}
Nov 01 14:29:17.100145 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.100063    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq" event={"ID":"b4f3576f-f0a4-4eb2-a770-6b52af66d488","Type":"ContainerDied","Data":"40e505b12afc66bc8a17d2dcc354685d109acfbd3d389ceee9fc037ade2f9018"}
Nov 01 14:29:17.100145 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.100097    5423 scope.go:122] "RemoveContainer" containerID="fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841"
Nov 01 14:29:17.112608 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.112586    5423 scope.go:122] "RemoveContainer" containerID="c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426"
Nov 01 14:29:17.120826 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.120800    5423 scope.go:122] "RemoveContainer" containerID="fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841"
Nov 01 14:29:17.121135 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:17.121112    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841\": container with ID starting with fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841 not found: ID does not exist" containerID="fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841"
Nov 01 14:29:17.121234 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.121148    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841"} err="failed to get container status \"fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841\": rpc error: code = NotFound desc = could not find container \"fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841\": container with ID starting with fac75d08b5556bdf2e77afd6039b3ecf4816e2519b5a508d56c77f195e92b841 not found: ID does not exist"
Nov 01 14:29:17.121234 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.121172    5423 scope.go:122] "RemoveContainer" containerID="c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426"
Nov 01 14:29:17.121601 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:17.121574    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426\": container with ID starting with c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426 not found: ID does not exist" containerID="c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426"
Nov 01 14:29:17.121707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.121609    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426"} err="failed to get container status \"c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426\": rpc error: code = NotFound desc = could not find container \"c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426\": container with ID starting with c856d10d788c3eb4392fd66de5f8645f282b8c3270c040ba99660f11cb53c426 not found: ID does not exist"
Nov 01 14:29:17.279604 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.279573    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"]
Nov 01 14:29:17.329887 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.329857    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d8f1c66f-openshift-default-6c47886d5-dzrrq"]
Nov 01 14:29:17.426185 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.426116    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"]
Nov 01 14:29:17.426562 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.426548    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy"
Nov 01 14:29:17.426605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.426563    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy"
Nov 01 14:29:17.426605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.426572    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy"
Nov 01 14:29:17.426605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.426577    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy"
Nov 01 14:29:17.426696 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.426642    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy"
Nov 01 14:29:17.426696 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.426650    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" containerName="istio-proxy"
Nov 01 14:29:17.431166 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.431144    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.494409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.493269    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5b0dcc-openshift-default-dockercfg-4d7r2\""
Nov 01 14:29:17.496093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.496065    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"]
Nov 01 14:29:17.535359 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535325    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.535548 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535370    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.535548 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535516    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.535635 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535571    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.535635 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535616    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d59485-1675-4de0-a87d-35fac5a4e730-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.535790 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535643    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpcxb\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-kube-api-access-cpcxb\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.535790 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535716    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.535790 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535782    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.535943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.535826    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637095    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637142    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637183    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637231    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637264    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d59485-1675-4de0-a87d-35fac5a4e730-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637298    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpcxb\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-kube-api-access-cpcxb\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637330    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637434    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637489    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637658    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.637838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.637681    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.638185 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.638157    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.638281 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.638228    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.638281 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.638255    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d59485-1675-4de0-a87d-35fac5a4e730-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.639889 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.639870    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.640053 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.640035    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.694856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.694783    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.694992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.694876    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpcxb\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-kube-api-access-cpcxb\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.743592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.743561    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:17.960568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:17.960498    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f3576f-f0a4-4eb2-a770-6b52af66d488" path="/var/lib/kubelet/pods/b4f3576f-f0a4-4eb2-a770-6b52af66d488/volumes"
Nov 01 14:29:18.181108 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:29:18.181082    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d59485_1675_4de0_a87d_35fac5a4e730.slice/crio-79d279b92712f03a20e38334c14062f36ee9357a04904dfdf83c43d333c4133c WatchSource:0}: Error finding container 79d279b92712f03a20e38334c14062f36ee9357a04904dfdf83c43d333c4133c: Status 404 returned error can't find the container with id 79d279b92712f03a20e38334c14062f36ee9357a04904dfdf83c43d333c4133c
Nov 01 14:29:18.182423 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:18.182395    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"]
Nov 01 14:29:18.183864 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:18.183835    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:18.183935 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:18.183898    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:18.183935 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:18.183931    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:19.110638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:19.110604    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" event={"ID":"f0d59485-1675-4de0-a87d-35fac5a4e730","Type":"ContainerStarted","Data":"66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950"}
Nov 01 14:29:19.110793 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:19.110663    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" event={"ID":"f0d59485-1675-4de0-a87d-35fac5a4e730","Type":"ContainerStarted","Data":"79d279b92712f03a20e38334c14062f36ee9357a04904dfdf83c43d333c4133c"}
Nov 01 14:29:19.280570 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:19.280524    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podStartSLOduration=2.280510511 podStartE2EDuration="2.280510511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:19.279370745 +0000 UTC m=+699.444856043" watchObservedRunningTime="2026-11-01 14:29:19.280510511 +0000 UTC m=+699.445995809"
Nov 01 14:29:19.743899 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:19.743863    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:19.749048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:19.749020    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:20.116201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:20.116167    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:20.117178 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:20.117154    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:22.605862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:22.605827    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"]
Nov 01 14:29:22.606321 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:22.606028    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy" containerID="cri-o://1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19" gracePeriod=30
Nov 01 14:29:27.863007 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.862979    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:29:27.944728 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944645    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-socket\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.944728 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944693    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5139751-71a0-4fd3-953c-99fdcd05fb44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5139751-71a0-4fd3-953c-99fdcd05fb44-istiod-ca-cert\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.944918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944772    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-token\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-token\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.944918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944801    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-podinfo\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.944918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944837    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-envoy\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.944918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944869    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-credential-socket\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.945141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944927    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-kube-api-access-thqc9\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-kube-api-access-thqc9\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.945141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944966    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-socket" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:27.945141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.944979    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-certs\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.945141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.945031    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-data\") pod \"a5139751-71a0-4fd3-953c-99fdcd05fb44\" (UID: \"a5139751-71a0-4fd3-953c-99fdcd05fb44\") "
Nov 01 14:29:27.945351 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.945245    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-credential-socket" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:27.945351 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.945273    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5139751-71a0-4fd3-953c-99fdcd05fb44-istiod-ca-cert" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:27.945490 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.945454    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-certs" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:27.945490 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.945466    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:27.945589 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.945508    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5139751-71a0-4fd3-953c-99fdcd05fb44-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:27.945589 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.945524    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:27.945589 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.945469    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-data" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:27.947019 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.946991    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-token" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:27.947137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.947044    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-envoy" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:29:27.947479 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.947460    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-kube-api-access-thqc9" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "kube-api-access-thqc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:27.947535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:27.947460    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-podinfo" pod "a5139751-71a0-4fd3-953c-99fdcd05fb44" (UID: "a5139751-71a0-4fd3-953c-99fdcd05fb44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:29:28.045978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.045950    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:28.045978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.045972    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:28.045978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.045983    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:28.046228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.045992    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thqc9\" (UniqueName: \"kubernetes.io/projected/a5139751-71a0-4fd3-953c-99fdcd05fb44-kube-api-access-thqc9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:28.046228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.046002    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:28.046228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.046009    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5139751-71a0-4fd3-953c-99fdcd05fb44-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:28.158545 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.158512    5423 generic.go:356] "Generic (PLEG): container finished" podID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerID="1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19" exitCode=0
Nov 01 14:29:28.158711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.158568    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"
Nov 01 14:29:28.158711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.158597    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" event={"ID":"a5139751-71a0-4fd3-953c-99fdcd05fb44","Type":"ContainerDied","Data":"1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19"}
Nov 01 14:29:28.158711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.158652    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc" event={"ID":"a5139751-71a0-4fd3-953c-99fdcd05fb44","Type":"ContainerDied","Data":"22c8c50c145b1d49d233ce185672c3d1f0a34cd3dcda14a281a184d8a0cd349b"}
Nov 01 14:29:28.158711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.158675    5423 scope.go:122] "RemoveContainer" containerID="1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19"
Nov 01 14:29:28.167823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.167806    5423 scope.go:122] "RemoveContainer" containerID="ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00"
Nov 01 14:29:28.175663 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.175634    5423 scope.go:122] "RemoveContainer" containerID="1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19"
Nov 01 14:29:28.175910 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:28.175890    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19\": container with ID starting with 1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19 not found: ID does not exist" containerID="1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19"
Nov 01 14:29:28.176012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.175922    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19"} err="failed to get container status \"1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19\": rpc error: code = NotFound desc = could not find container \"1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19\": container with ID starting with 1ccbdbc83a9d30fce823158b5c82f6e4e03f67a93822136330a30dec9d908d19 not found: ID does not exist"
Nov 01 14:29:28.176012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.175945    5423 scope.go:122] "RemoveContainer" containerID="ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00"
Nov 01 14:29:28.176219 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:28.176202    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00\": container with ID starting with ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00 not found: ID does not exist" containerID="ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00"
Nov 01 14:29:28.176266 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.176229    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00"} err="failed to get container status \"ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00\": rpc error: code = NotFound desc = could not find container \"ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00\": container with ID starting with ae248c8a203f145c4053e74059e1b5bc75bced1323e7ee8cbc96c509a6aeef00 not found: ID does not exist"
Nov 01 14:29:28.322401 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.322357    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"]
Nov 01 14:29:28.325806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:28.325785    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f7c71759-openshift-default-54785b7659-62xsc"]
Nov 01 14:29:29.716794 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.716756    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"]
Nov 01 14:29:29.717232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.717218    5423 cpu_manager.go:414] "containerMap: removing container" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy"
Nov 01 14:29:29.717276 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.717233    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy"
Nov 01 14:29:29.717313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.717307    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy"
Nov 01 14:29:29.721991 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.721972    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:29.761124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.761093    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4\" (UID: \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:29.761124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.761124    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljrk\" (UniqueName: \"kubernetes.io/projected/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-kube-api-access-fljrk\") pod \"payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4\" (UID: \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:29.773544 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.773520    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"]
Nov 01 14:29:29.821677 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.821657    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5b0dcc\""
Nov 01 14:29:29.821815 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.821681    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5b0dcc-dockercfg-9s7n7\""
Nov 01 14:29:29.861987 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.861961    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4\" (UID: \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:29.862107 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.861993    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljrk\" (UniqueName: \"kubernetes.io/projected/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-kube-api-access-fljrk\") pod \"payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4\" (UID: \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:29.862748 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.862731    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4\" (UID: \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:29.918821 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.918802    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljrk\" (UniqueName: \"kubernetes.io/projected/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-kube-api-access-fljrk\") pod \"payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4\" (UID: \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:29.958730 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:29.958704    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" path="/var/lib/kubelet/pods/a5139751-71a0-4fd3-953c-99fdcd05fb44/volumes"
Nov 01 14:29:30.035287 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.035228    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:30.054655 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.054629    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"]
Nov 01 14:29:30.055157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.055140    5423 cpu_manager.go:414] "containerMap: removing container" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy"
Nov 01 14:29:30.055157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.055158    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy"
Nov 01 14:29:30.055269 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.055234    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5139751-71a0-4fd3-953c-99fdcd05fb44" containerName="istio-proxy"
Nov 01 14:29:30.058301 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.058284    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.111221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.111194    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-5b0dcc-serving-cert\""
Nov 01 14:29:30.111373 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.111355    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"]
Nov 01 14:29:30.164875 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.164847    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.165005 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.164883    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610938fd-7cd7-475c-b75a-c095c335fc7a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.165005 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.164961    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxf95\" (UniqueName: \"kubernetes.io/projected/610938fd-7cd7-475c-b75a-c095c335fc7a-kube-api-access-rxf95\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.165005 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.164986    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-metrics-certs\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.266052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.266018    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.266214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.266062    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610938fd-7cd7-475c-b75a-c095c335fc7a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.266214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.266108    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxf95\" (UniqueName: \"kubernetes.io/projected/610938fd-7cd7-475c-b75a-c095c335fc7a-kube-api-access-rxf95\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.266214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.266147    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-metrics-certs\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.266214 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:30.266170    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-serving-cert: secret "maas-api-e2e-worker-w3-5b0dcc-serving-cert" not found
Nov 01 14:29:30.266486 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:30.266242    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls podName:610938fd-7cd7-475c-b75a-c095c335fc7a nodeName:}" failed. No retries permitted until 2026-11-01 14:29:30.766223481 +0000 UTC m=+710.931708770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls") pod "maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c" (UID: "610938fd-7cd7-475c-b75a-c095c335fc7a") : secret "maas-api-e2e-worker-w3-5b0dcc-serving-cert" not found
Nov 01 14:29:30.266813 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.266794    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610938fd-7cd7-475c-b75a-c095c335fc7a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.268624 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.268607    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-metrics-certs\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.325931 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.325897    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxf95\" (UniqueName: \"kubernetes.io/projected/610938fd-7cd7-475c-b75a-c095c335fc7a-kube-api-access-rxf95\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.401000 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:29:30.400971    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53047b0f_dbe2_4447_874d_7b6d47cbf3fa.slice/crio-c53e4c7b25ed998b27f61597b2125421912abd728405865cc3d7c6aeb59a0444 WatchSource:0}: Error finding container c53e4c7b25ed998b27f61597b2125421912abd728405865cc3d7c6aeb59a0444: Status 404 returned error can't find the container with id c53e4c7b25ed998b27f61597b2125421912abd728405865cc3d7c6aeb59a0444
Nov 01 14:29:30.402245 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.402220    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"]
Nov 01 14:29:30.773634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.773548    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.775932 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.775908    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:30.972853 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:30.972821    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:31.175973 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:31.175937    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4" event={"ID":"53047b0f-dbe2-4447-874d-7b6d47cbf3fa","Type":"ContainerStarted","Data":"0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2"}
Nov 01 14:29:31.176164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:31.175985    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4" event={"ID":"53047b0f-dbe2-4447-874d-7b6d47cbf3fa","Type":"ContainerStarted","Data":"c53e4c7b25ed998b27f61597b2125421912abd728405865cc3d7c6aeb59a0444"}
Nov 01 14:29:31.176164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:31.176018    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:31.291856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:31.291762    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4" podStartSLOduration=2.291747667 podStartE2EDuration="2.291747667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:31.290342471 +0000 UTC m=+711.455827761" watchObservedRunningTime="2026-11-01 14:29:31.291747667 +0000 UTC m=+711.457232965"
Nov 01 14:29:31.385330 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:29:31.385297    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610938fd_7cd7_475c_b75a_c095c335fc7a.slice/crio-0b39aee01e6828daff51429d89977e6d21b2e0cd51a7ab764a2ad7882da96550 WatchSource:0}: Error finding container 0b39aee01e6828daff51429d89977e6d21b2e0cd51a7ab764a2ad7882da96550: Status 404 returned error can't find the container with id 0b39aee01e6828daff51429d89977e6d21b2e0cd51a7ab764a2ad7882da96550
Nov 01 14:29:31.385932 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:31.385907    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"]
Nov 01 14:29:32.182608 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:32.182575    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c" event={"ID":"610938fd-7cd7-475c-b75a-c095c335fc7a","Type":"ContainerStarted","Data":"0b39aee01e6828daff51429d89977e6d21b2e0cd51a7ab764a2ad7882da96550"}
Nov 01 14:29:32.898348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:32.898314    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:29:33.189238 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:33.189166    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c" event={"ID":"610938fd-7cd7-475c-b75a-c095c335fc7a","Type":"ContainerStarted","Data":"a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a"}
Nov 01 14:29:33.189238 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:33.189232    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:33.315794 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:33.315742    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c" podStartSLOduration=2.4094305719999998 podStartE2EDuration="3.31572751s" totalImagesPullingTime="906.296938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:33.314010264 +0000 UTC m=+713.479495564" watchObservedRunningTime="2026-11-01 14:29:33.31572751 +0000 UTC m=+713.481212805"
Nov 01 14:29:35.149604 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:35.149569    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:29:39.216732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:39.216688    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 14:29:39.392355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:39.392325    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:29:42.186214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:42.186177    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 14:29:43.265510 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:43.265476    5423 generic.go:356] "Generic (PLEG): container finished" podID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerID="ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57" exitCode=0
Nov 01 14:29:43.265938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:43.265547    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" event={"ID":"61f5dd4e-c04a-47ec-8291-5a6565e93bde","Type":"ContainerDied","Data":"ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57"}
Nov 01 14:29:43.265938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:43.265924    5423 scope.go:122] "RemoveContainer" containerID="ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57"
Nov 01 14:29:43.267241 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:43.267216    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:43.267312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:43.267280    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:43.267353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:43.267314    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:44.272381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:44.272350    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" event={"ID":"61f5dd4e-c04a-47ec-8291-5a6565e93bde","Type":"ContainerStarted","Data":"3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f"}
Nov 01 14:29:45.135096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:45.135067    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:45.146442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:45.146407    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 14:29:45.148586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:45.146687    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 14:29:46.080052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:46.080009    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:46.134509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:46.134403    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 14:29:46.134509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:46.134473    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 14:29:47.127891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:47.127448    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 14:29:47.127891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:47.127521    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 14:29:48.127410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:48.126729    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 14:29:48.127410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:48.126796    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 14:29:49.130165 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.129731    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 01 14:29:49.130165 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.129793    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 01 14:29:49.665772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.665728    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"]
Nov 01 14:29:49.670772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.670747    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.775128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.775085    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-metrics-certs\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.775128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.775129    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8st4b\" (UniqueName: \"kubernetes.io/projected/fe198337-d3c3-4aed-a9d3-8457fcfed95c-kube-api-access-8st4b\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.775338 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.775280    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe198337-d3c3-4aed-a9d3-8457fcfed95c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.775338 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.775326    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.776037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.776013    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-536a87-serving-cert\""
Nov 01 14:29:49.777688 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.777666    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"]
Nov 01 14:29:49.875918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.875883    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-metrics-certs\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.875918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.875921    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8st4b\" (UniqueName: \"kubernetes.io/projected/fe198337-d3c3-4aed-a9d3-8457fcfed95c-kube-api-access-8st4b\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.876141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.875980    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe198337-d3c3-4aed-a9d3-8457fcfed95c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.876141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.876008    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.876254 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:49.876189    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-serving-cert: secret "maas-api-e2e-worker-w4-536a87-serving-cert" not found
Nov 01 14:29:49.876305 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:29:49.876268    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls podName:fe198337-d3c3-4aed-a9d3-8457fcfed95c nodeName:}" failed. No retries permitted until 2026-11-01 14:29:50.376245463 +0000 UTC m=+730.541730754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls") pod "maas-api-e2e-worker-w4-536a87-69785c9797-gskvk" (UID: "fe198337-d3c3-4aed-a9d3-8457fcfed95c") : secret "maas-api-e2e-worker-w4-536a87-serving-cert" not found
Nov 01 14:29:49.876891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.876869    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe198337-d3c3-4aed-a9d3-8457fcfed95c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.879295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.879270    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-metrics-certs\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:49.938940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:49.938862    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8st4b\" (UniqueName: \"kubernetes.io/projected/fe198337-d3c3-4aed-a9d3-8457fcfed95c-kube-api-access-8st4b\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:50.129056 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:50.129021    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:50.129994 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:50.129971    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 14:29:50.380181 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:50.380130    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:50.383226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:50.382703    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-536a87-69785c9797-gskvk\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:50.585477 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:50.585446    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:50.981891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:50.981855    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": read tcp 10.133.0.2:54654->10.133.0.49:15021: read: connection reset by peer" start-of-body=
Nov 01 14:29:50.982062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:50.981917    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": read tcp 10.133.0.2:54654->10.133.0.49:15021: read: connection reset by peer"
Nov 01 14:29:50.985926 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:29:50.985846    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe198337_d3c3_4aed_a9d3_8457fcfed95c.slice/crio-cdee74c43963e461eb46d714df301a83978bf5b22dae4bb99e79aaa66fdd5a03 WatchSource:0}: Error finding container cdee74c43963e461eb46d714df301a83978bf5b22dae4bb99e79aaa66fdd5a03: Status 404 returned error can't find the container with id cdee74c43963e461eb46d714df301a83978bf5b22dae4bb99e79aaa66fdd5a03
Nov 01 14:29:50.986710 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:50.986685    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"]
Nov 01 14:29:51.353120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:51.353075    5423 generic.go:356] "Generic (PLEG): container finished" podID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerID="66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950" exitCode=0
Nov 01 14:29:51.353294 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:51.353169    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" event={"ID":"f0d59485-1675-4de0-a87d-35fac5a4e730","Type":"ContainerDied","Data":"66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950"}
Nov 01 14:29:51.353610 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:51.353593    5423 scope.go:122] "RemoveContainer" containerID="66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950"
Nov 01 14:29:51.356525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:51.355916    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:51.356525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:51.356039    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:51.356525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:51.356090    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:51.358188 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:51.357899    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk" event={"ID":"fe198337-d3c3-4aed-a9d3-8457fcfed95c","Type":"ContainerStarted","Data":"cdee74c43963e461eb46d714df301a83978bf5b22dae4bb99e79aaa66fdd5a03"}
Nov 01 14:29:52.366475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:52.365850    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" event={"ID":"f0d59485-1675-4de0-a87d-35fac5a4e730","Type":"ContainerStarted","Data":"7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a"}
Nov 01 14:29:52.368280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:52.368224    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk" event={"ID":"fe198337-d3c3-4aed-a9d3-8457fcfed95c","Type":"ContainerStarted","Data":"2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b"}
Nov 01 14:29:52.368425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:52.368292    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:52.744872 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:52.744838    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:52.746123 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:52.746090    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 14:29:52.746261 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:52.746149    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 14:29:53.746284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:53.746055    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 14:29:53.746284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:53.746121    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 14:29:54.744118 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:54.744080    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 14:29:54.744293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:54.744145    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 14:29:55.103513 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:55.089803    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk" podStartSLOduration=5.124072545 podStartE2EDuration="6.089781063s" totalImagesPullingTime="965.708518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:52.618070514 +0000 UTC m=+732.783555814" watchObservedRunningTime="2026-11-01 14:29:55.089781063 +0000 UTC m=+735.255266364"
Nov 01 14:29:55.103513 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:55.090550    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:29:55.744794 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:55.744756    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 14:29:55.744964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:55.744822    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 14:29:56.390732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:56.390708    5423 generic.go:356] "Generic (PLEG): container finished" podID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerID="e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20" exitCode=0
Nov 01 14:29:56.391072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:56.390774    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" event={"ID":"7dda68f1-989b-4643-9013-0e2962c9b4d9","Type":"ContainerDied","Data":"e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20"}
Nov 01 14:29:56.391147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:56.391132    5423 scope.go:122] "RemoveContainer" containerID="e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20"
Nov 01 14:29:56.392348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:56.392322    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:56.392466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:56.392450    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:56.392529 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:56.392498    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:29:56.744242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:56.744158    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 14:29:56.744242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:56.744221    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 14:29:57.409743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:57.409700    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" event={"ID":"7dda68f1-989b-4643-9013-0e2962c9b4d9","Type":"ContainerStarted","Data":"ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6"}
Nov 01 14:29:57.747929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:57.747855    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:57.748162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:57.748139    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:57.748891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:57.748871    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 14:29:58.319439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:58.316555    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:29:58.319439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:58.318288    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 14:29:58.319439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:58.318339    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 14:29:58.378172 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:58.378149    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 14:29:59.315409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:59.314774    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 14:29:59.315409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:29:59.314851    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 14:30:00.075574 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.075538    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:30:00.178525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.178494    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"]
Nov 01 14:30:00.182061 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.182046    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:00.234701 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.234669    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 14:30:00.277788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.277759    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4\" (UID: \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:00.277940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.277815    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tslpm\" (UniqueName: \"kubernetes.io/projected/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-kube-api-access-tslpm\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4\" (UID: \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:00.286720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.286691    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"]
Nov 01 14:30:00.319831 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.319792    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 14:30:00.320321 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.319848    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 14:30:00.379454 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.379352    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4\" (UID: \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:00.379454 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.379439    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tslpm\" (UniqueName: \"kubernetes.io/projected/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-kube-api-access-tslpm\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4\" (UID: \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:00.380050 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.380029    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4\" (UID: \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:00.438535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.438510    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tslpm\" (UniqueName: \"kubernetes.io/projected/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-kube-api-access-tslpm\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4\" (UID: \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:00.516847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.516545    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:00.883362 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:30:00.883327    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4638ecc_00fd_49bf_a20f_5c24eb8222dd.slice/crio-f3ae18563004efc8c18d49e0a4bb3779f8369bb55007402863ad59f4933e7c2d WatchSource:0}: Error finding container f3ae18563004efc8c18d49e0a4bb3779f8369bb55007402863ad59f4933e7c2d: Status 404 returned error can't find the container with id f3ae18563004efc8c18d49e0a4bb3779f8369bb55007402863ad59f4933e7c2d
Nov 01 14:30:00.884417 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:00.884368    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"]
Nov 01 14:30:01.315409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:01.315277    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 14:30:01.315409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:01.315353    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 14:30:01.437458 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:01.437417    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" event={"ID":"d4638ecc-00fd-49bf-a20f-5c24eb8222dd","Type":"ContainerStarted","Data":"f3ae18563004efc8c18d49e0a4bb3779f8369bb55007402863ad59f4933e7c2d"}
Nov 01 14:30:02.313766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:02.313730    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 14:30:02.314048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:02.314019    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 14:30:03.318120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:03.318091    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:30:03.319188 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:03.319169    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:30:04.581603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:04.580929    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:30:05.164649 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:30:05.164613    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:30:06.485592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:06.485559    5423 generic.go:356] "Generic (PLEG): container finished" podID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerID="f4b1f703eeb00291bc5fd39aa3fc5faf428858ec693e5883403770648b8f2ba3" exitCode=60
Nov 01 14:30:06.485592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:06.485603    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" event={"ID":"d4638ecc-00fd-49bf-a20f-5c24eb8222dd","Type":"ContainerDied","Data":"f4b1f703eeb00291bc5fd39aa3fc5faf428858ec693e5883403770648b8f2ba3"}
Nov 01 14:30:06.485991 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:06.485828    5423 scope.go:122] "RemoveContainer" containerID="f4b1f703eeb00291bc5fd39aa3fc5faf428858ec693e5883403770648b8f2ba3"
Nov 01 14:30:07.493233 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:07.493202    5423 generic.go:356] "Generic (PLEG): container finished" podID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerID="a3e4662d6d4f8160117696afb1a5ea422ab0111c8ca26a6f326066c4bd17ad3e" exitCode=60
Nov 01 14:30:07.493669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:07.493286    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" event={"ID":"d4638ecc-00fd-49bf-a20f-5c24eb8222dd","Type":"ContainerDied","Data":"a3e4662d6d4f8160117696afb1a5ea422ab0111c8ca26a6f326066c4bd17ad3e"}
Nov 01 14:30:07.493669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:07.493343    5423 scope.go:122] "RemoveContainer" containerID="f4b1f703eeb00291bc5fd39aa3fc5faf428858ec693e5883403770648b8f2ba3"
Nov 01 14:30:07.493669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:07.493601    5423 scope.go:122] "RemoveContainer" containerID="a3e4662d6d4f8160117696afb1a5ea422ab0111c8ca26a6f326066c4bd17ad3e"
Nov 01 14:30:07.493846 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:30:07.493824    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4_odh-ai-gateway-infra(d4638ecc-00fd-49bf-a20f-5c24eb8222dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd"
Nov 01 14:30:08.502618 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:08.502590    5423 scope.go:122] "RemoveContainer" containerID="a3e4662d6d4f8160117696afb1a5ea422ab0111c8ca26a6f326066c4bd17ad3e"
Nov 01 14:30:08.503009 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:30:08.502860    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4_odh-ai-gateway-infra(d4638ecc-00fd-49bf-a20f-5c24eb8222dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd"
Nov 01 14:30:19.956117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:19.956088    5423 scope.go:122] "RemoveContainer" containerID="a3e4662d6d4f8160117696afb1a5ea422ab0111c8ca26a6f326066c4bd17ad3e"
Nov 01 14:30:20.563738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:20.563704    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" event={"ID":"d4638ecc-00fd-49bf-a20f-5c24eb8222dd","Type":"ContainerStarted","Data":"2003d608059d8fddab4ba59eef556a8b4c62ce6380d17c59b8a1e75d8d7f0144"}
Nov 01 14:30:20.686203 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:20.686150    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" podStartSLOduration=16.107011445 podStartE2EDuration="20.686138452s" totalImagesPullingTime="4.579127007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:30:20.685374179 +0000 UTC m=+760.850859476" watchObservedRunningTime="2026-11-01 14:30:20.686138452 +0000 UTC m=+760.851623814"
Nov 01 14:30:21.570826 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:21.570788    5423 generic.go:356] "Generic (PLEG): container finished" podID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerID="2003d608059d8fddab4ba59eef556a8b4c62ce6380d17c59b8a1e75d8d7f0144" exitCode=60
Nov 01 14:30:21.571246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:21.570862    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" event={"ID":"d4638ecc-00fd-49bf-a20f-5c24eb8222dd","Type":"ContainerDied","Data":"2003d608059d8fddab4ba59eef556a8b4c62ce6380d17c59b8a1e75d8d7f0144"}
Nov 01 14:30:21.571246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:21.570928    5423 scope.go:122] "RemoveContainer" containerID="a3e4662d6d4f8160117696afb1a5ea422ab0111c8ca26a6f326066c4bd17ad3e"
Nov 01 14:30:21.571335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:21.571294    5423 scope.go:122] "RemoveContainer" containerID="2003d608059d8fddab4ba59eef556a8b4c62ce6380d17c59b8a1e75d8d7f0144"
Nov 01 14:30:21.571605 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:30:21.571579    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4_odh-ai-gateway-infra(d4638ecc-00fd-49bf-a20f-5c24eb8222dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd"
Nov 01 14:30:21.694274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:21.694248    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"]
Nov 01 14:30:22.745164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:22.745138    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:22.844469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:22.844436    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-kube-api-access-tslpm\" (UniqueName: \"kubernetes.io/projected/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-kube-api-access-tslpm\") pod \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\" (UID: \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\") "
Nov 01 14:30:22.844660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:22.844557    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-maas-api-service-ca\") pod \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\" (UID: \"d4638ecc-00fd-49bf-a20f-5c24eb8222dd\") "
Nov 01 14:30:22.844917 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:22.844894    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-maas-api-service-ca" pod "d4638ecc-00fd-49bf-a20f-5c24eb8222dd" (UID: "d4638ecc-00fd-49bf-a20f-5c24eb8222dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:30:22.846525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:22.846506    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-kube-api-access-tslpm" pod "d4638ecc-00fd-49bf-a20f-5c24eb8222dd" (UID: "d4638ecc-00fd-49bf-a20f-5c24eb8222dd"). InnerVolumeSpecName "kube-api-access-tslpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:30:22.945616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:22.945587    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:30:22.945616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:22.945613    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tslpm\" (UniqueName: \"kubernetes.io/projected/d4638ecc-00fd-49bf-a20f-5c24eb8222dd-kube-api-access-tslpm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:30:23.607364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:23.607337    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"
Nov 01 14:30:23.607561 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:23.607370    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4" event={"ID":"d4638ecc-00fd-49bf-a20f-5c24eb8222dd","Type":"ContainerDied","Data":"f3ae18563004efc8c18d49e0a4bb3779f8369bb55007402863ad59f4933e7c2d"}
Nov 01 14:30:23.607561 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:23.607449    5423 scope.go:122] "RemoveContainer" containerID="2003d608059d8fddab4ba59eef556a8b4c62ce6380d17c59b8a1e75d8d7f0144"
Nov 01 14:30:23.772819 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:23.772787    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"]
Nov 01 14:30:23.823036 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:23.823007    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892390-74bd4"]
Nov 01 14:30:23.959887 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:23.959810    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" path="/var/lib/kubelet/pods/d4638ecc-00fd-49bf-a20f-5c24eb8222dd/volumes"
Nov 01 14:30:30.765422 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:30.765370    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:30:35.148875 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:30:35.148836    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:30:47.544323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:47.544290    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:30:56.860664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.860634    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"]
Nov 01 14:30:56.861082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.861039    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:30:56.861082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.861050    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:30:56.861082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.861074    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:30:56.861082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.861080    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:30:56.861211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.861145    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:30:56.861211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.861158    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:30:56.861211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.861165    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:30:56.864445 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.864429    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:56.917281 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.917252    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"]
Nov 01 14:30:56.965930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.965896    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"kube-root-ca.crt\""
Nov 01 14:30:56.965930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.965915    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"openshift-service-ca.crt\""
Nov 01 14:30:56.966173 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.966011    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"default-dockercfg-glbls\""
Nov 01 14:30:56.967442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.967425    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"simulator-w4-536a87-kserve-self-signed-certs\""
Nov 01 14:30:56.998845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.997999    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-dshm\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:56.998845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.998137    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-model-cache\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:56.998845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.998244    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tmp-dir\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:56.998845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.998274    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-home\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:56.998845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.998343    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tls-certs\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:56.998845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:56.998378    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsn5\" (UniqueName: \"kubernetes.io/projected/29709b9a-d3f9-4795-a58a-9b2ba35950fb-kube-api-access-nxsn5\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.099716 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.099683    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-model-cache\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.099896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.099740    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tmp-dir\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.099896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.099768    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-home\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.099896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.099796    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tls-certs\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.099896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.099815    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsn5\" (UniqueName: \"kubernetes.io/projected/29709b9a-d3f9-4795-a58a-9b2ba35950fb-kube-api-access-nxsn5\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.099896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.099890    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-dshm\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.100176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.100128    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-model-cache\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.100176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.100170    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-home\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.100448 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.100178    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tmp-dir\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.102323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.102301    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-dshm\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.102770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.102753    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tls-certs\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.157965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.157897    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsn5\" (UniqueName: \"kubernetes.io/projected/29709b9a-d3f9-4795-a58a-9b2ba35950fb-kube-api-access-nxsn5\") pod \"simulator-w4-536a87-kserve-7ff878867-k58k2\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") " pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.176833 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.176802    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:30:57.542602 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:30:57.542568    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29709b9a_d3f9_4795_a58a_9b2ba35950fb.slice/crio-5c8a3714e88d5d7888b4e4c413e5fc08b87283e9449dda6d8135a4b6af240632 WatchSource:0}: Error finding container 5c8a3714e88d5d7888b4e4c413e5fc08b87283e9449dda6d8135a4b6af240632: Status 404 returned error can't find the container with id 5c8a3714e88d5d7888b4e4c413e5fc08b87283e9449dda6d8135a4b6af240632
Nov 01 14:30:57.544214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.544194    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"]
Nov 01 14:30:57.778507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.778472    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" event={"ID":"29709b9a-d3f9-4795-a58a-9b2ba35950fb","Type":"ContainerStarted","Data":"70f45ac23fc38002775042400ec16e6fc2b8c4943cec90791b71d61ee23a2e82"}
Nov 01 14:30:57.778687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.778520    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" event={"ID":"29709b9a-d3f9-4795-a58a-9b2ba35950fb","Type":"ContainerStarted","Data":"5c8a3714e88d5d7888b4e4c413e5fc08b87283e9449dda6d8135a4b6af240632"}
Nov 01 14:30:57.893771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:57.893721    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podStartSLOduration=1.893708657 podStartE2EDuration="1.893708657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:30:57.892266185 +0000 UTC m=+798.057751483" watchObservedRunningTime="2026-11-01 14:30:57.893708657 +0000 UTC m=+798.059193988"
Nov 01 14:30:58.176942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:30:58.176868    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:31:04.870189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:04.870153    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:31:05.150030 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:31:05.149946    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:31:07.177641 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:07.177603    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:31:07.190328 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:07.190299    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:31:07.838020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:07.837993    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:31:15.564082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.564045    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"]
Nov 01 14:31:15.564503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.564489    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:31:15.564503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.564503    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4638ecc-00fd-49bf-a20f-5c24eb8222dd" containerName="cleanup"
Nov 01 14:31:15.567984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.567969    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.620745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.620699    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"]
Nov 01 14:31:15.668138 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.668113    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"kube-root-ca.crt\""
Nov 01 14:31:15.668264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.668117    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"openshift-service-ca.crt\""
Nov 01 14:31:15.668264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.668212    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"default-dockercfg-nqggd\""
Nov 01 14:31:15.670360 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.670342    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"premium-w3-5b0dcc-kserve-self-signed-certs\""
Nov 01 14:31:15.680188 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.680169    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-tmp-dir\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.680276 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.680210    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd534096-6aae-4990-b15c-bb5d262857a6-tls-certs\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.680317 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.680273    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-model-cache\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.680350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.680324    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrgnl\" (UniqueName: \"kubernetes.io/projected/fd534096-6aae-4990-b15c-bb5d262857a6-kube-api-access-lrgnl\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.680350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.680346    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-home\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.680449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.680360    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-dshm\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.781658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.781623    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-tmp-dir\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.781821 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.781674    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd534096-6aae-4990-b15c-bb5d262857a6-tls-certs\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.781821 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.781712    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-model-cache\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.781821 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.781775    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgnl\" (UniqueName: \"kubernetes.io/projected/fd534096-6aae-4990-b15c-bb5d262857a6-kube-api-access-lrgnl\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.781821 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.781805    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-home\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.782024 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.781825    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-dshm\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.782137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.782111    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-tmp-dir\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.782137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.782128    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-model-cache\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.782229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.782152    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-home\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.784029 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.784005    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-dshm\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.784269 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.784252    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd534096-6aae-4990-b15c-bb5d262857a6-tls-certs\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.838354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.838286    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgnl\" (UniqueName: \"kubernetes.io/projected/fd534096-6aae-4990-b15c-bb5d262857a6-kube-api-access-lrgnl\") pod \"premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") " pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:15.903091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:15.903052    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:16.291889 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:31:16.291856    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd534096_6aae_4990_b15c_bb5d262857a6.slice/crio-2f0bb24738b78c4bbc73c54c00e700925f448cf2b9f998dbb73d83a015ed2512 WatchSource:0}: Error finding container 2f0bb24738b78c4bbc73c54c00e700925f448cf2b9f998dbb73d83a015ed2512: Status 404 returned error can't find the container with id 2f0bb24738b78c4bbc73c54c00e700925f448cf2b9f998dbb73d83a015ed2512
Nov 01 14:31:16.292916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:16.292895    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"]
Nov 01 14:31:16.873027 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:16.872989    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" event={"ID":"fd534096-6aae-4990-b15c-bb5d262857a6","Type":"ContainerStarted","Data":"5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656"}
Nov 01 14:31:16.873434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:16.873036    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" event={"ID":"fd534096-6aae-4990-b15c-bb5d262857a6","Type":"ContainerStarted","Data":"2f0bb24738b78c4bbc73c54c00e700925f448cf2b9f998dbb73d83a015ed2512"}
Nov 01 14:31:16.903301 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:16.903276    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:16.988186 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:16.988129    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podStartSLOduration=1.9881143300000002 podStartE2EDuration="1.98811433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:31:16.986683285 +0000 UTC m=+817.152168582" watchObservedRunningTime="2026-11-01 14:31:16.98811433 +0000 UTC m=+817.153599627"
Nov 01 14:31:23.108738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:23.108642    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:31:25.904064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:25.904020    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:25.916867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:25.916841    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:25.927967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:25.927941    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:31:35.148030 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:31:35.147996    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:31:49.019836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:31:49.019800    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:32:05.149990 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:32:05.149953    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:32:17.650881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:17.650841    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:32:35.148879 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:32:35.148833    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:32:36.918636 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:36.918606    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:32:40.059401 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:40.059347    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:32:40.061156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:40.061132    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:32:47.487943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:47.487907    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"]
Nov 01 14:32:47.488421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:47.488093    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy" containerID="cri-o://f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88" gracePeriod=30
Nov 01 14:32:52.740379 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.740347    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:32:52.864515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864484    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-data\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.864700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864548    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istiod-ca-cert\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.864700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864585    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-kube-api-access-9hw82\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-kube-api-access-9hw82\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.864700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864662    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-envoy\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.864883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864709    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-credential-socket\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.864883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864734    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-token\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-token\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.864883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864751    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-podinfo\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.864883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864767    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-data" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:32:52.864883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864820    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-socket\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.864883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.864855    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-certs\") pod \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\" (UID: \"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24\") "
Nov 01 14:32:52.865236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.865019    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-credential-socket" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:32:52.865236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.865032    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istiod-ca-cert" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:32:52.865236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.865207    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-certs" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:32:52.865415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.865324    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:52.865415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.865333    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-socket" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:32:52.865415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.865345    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:52.865415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.865377    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:52.865634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.865416    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:52.866929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.866902    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-kube-api-access-9hw82" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "kube-api-access-9hw82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:32:52.866929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.866916    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-podinfo" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:32:52.867291 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.867273    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-envoy" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:32:52.867335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.867310    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-token" pod "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" (UID: "20b9ce89-85c6-4df7-a6d8-a1bdbb858b24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:32:52.966465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.966428    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hw82\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-kube-api-access-9hw82\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:52.966465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.966457    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:52.966465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.966468    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:52.966729 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.966480    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:52.966729 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:52.966493    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:32:53.379858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.379828    5423 generic.go:356] "Generic (PLEG): container finished" podID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerID="f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88" exitCode=0
Nov 01 14:32:53.380027 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.379875    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"
Nov 01 14:32:53.380027 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.379904    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" event={"ID":"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24","Type":"ContainerDied","Data":"f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88"}
Nov 01 14:32:53.380027 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.379947    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp" event={"ID":"20b9ce89-85c6-4df7-a6d8-a1bdbb858b24","Type":"ContainerDied","Data":"1492cdd24150aebfd50a6022875a3de7db44bea46d1516121460e8ca953c706b"}
Nov 01 14:32:53.380027 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.379969    5423 scope.go:122] "RemoveContainer" containerID="f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88"
Nov 01 14:32:53.395121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.395097    5423 scope.go:122] "RemoveContainer" containerID="f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88"
Nov 01 14:32:53.395407 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:32:53.395363    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88\": container with ID starting with f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88 not found: ID does not exist" containerID="f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88"
Nov 01 14:32:53.395475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.395413    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88"} err="failed to get container status \"f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88\": rpc error: code = NotFound desc = could not find container \"f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88\": container with ID starting with f4b097b119913d4a80b77f183636e3a784fbf684816f7d733f59fd0f671d4e88 not found: ID does not exist"
Nov 01 14:32:53.595833 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.595803    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"]
Nov 01 14:32:53.645926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.645861    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rt5lp"]
Nov 01 14:32:53.960140 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:53.960070    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" path="/var/lib/kubelet/pods/20b9ce89-85c6-4df7-a6d8-a1bdbb858b24/volumes"
Nov 01 14:32:55.783311 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.783279    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"]
Nov 01 14:32:55.783747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.783733    5423 cpu_manager.go:414] "containerMap: removing container" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy"
Nov 01 14:32:55.783794 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.783747    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy"
Nov 01 14:32:55.783846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.783837    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b9ce89-85c6-4df7-a6d8-a1bdbb858b24" containerName="istio-proxy"
Nov 01 14:32:55.786909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.786877    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.839413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.839357    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-536a87-openshift-default-dockercfg-588rv\""
Nov 01 14:32:55.892586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.892549    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"]
Nov 01 14:32:55.901286 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.901232    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.901480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.901360    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.901598 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.901431    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbph\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-kube-api-access-fsbph\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.901696 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.901619    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.901785 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.901757    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.901990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.901973    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.902128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.902105    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.902240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.902141    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:55.902240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:55.902197    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.002923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.002893    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.002952    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.002997    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbph\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-kube-api-access-fsbph\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003028    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003058    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003113    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003154    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003180    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003238    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003309    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003905 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003881    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.003993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.003917    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.004444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.004422    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.004995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.004742    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.007021 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.007000    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.011880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.011857    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.062431 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.062404    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.062686 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.062662    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbph\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-kube-api-access-fsbph\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.104091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.100366    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:56.547604 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:32:56.547571    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecd34a5_f6a3_4e5d_91c2_02663eb01d72.slice/crio-148630c6f31be484ad900451f3ebadbb0ffd177cd1d493861e232c796bfdd5a2 WatchSource:0}: Error finding container 148630c6f31be484ad900451f3ebadbb0ffd177cd1d493861e232c796bfdd5a2: Status 404 returned error can't find the container with id 148630c6f31be484ad900451f3ebadbb0ffd177cd1d493861e232c796bfdd5a2
Nov 01 14:32:56.548658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.548628    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"]
Nov 01 14:32:56.549895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.549866    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:32:56.550004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.549928    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:32:56.550004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:56.549958    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:32:57.436472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:57.436425    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" event={"ID":"fecd34a5-f6a3-4e5d-91c2-02663eb01d72","Type":"ContainerStarted","Data":"04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3"}
Nov 01 14:32:57.436903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:57.436489    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" event={"ID":"fecd34a5-f6a3-4e5d-91c2-02663eb01d72","Type":"ContainerStarted","Data":"148630c6f31be484ad900451f3ebadbb0ffd177cd1d493861e232c796bfdd5a2"}
Nov 01 14:32:57.660200 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:57.660149    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" podStartSLOduration=2.660136257 podStartE2EDuration="2.660136257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:57.659920563 +0000 UTC m=+917.825405861" watchObservedRunningTime="2026-11-01 14:32:57.660136257 +0000 UTC m=+917.825621553"
Nov 01 14:32:58.113410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:58.108322    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:32:58.113410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:58.108456    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 14:32:58.113410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:58.108502    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 14:32:59.101826 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:59.101787    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 14:32:59.102438 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:32:59.102375    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 14:33:00.118411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:00.110788    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 14:33:00.118411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:00.110851    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 14:33:01.102838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:01.102798    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 14:33:01.103140 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:01.103111    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 14:33:02.110411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:02.104738    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 14:33:02.110411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:02.104796    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 14:33:03.218250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:03.218219    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:33:03.219529 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:03.219327    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:33:03.221920 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:03.221894    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 14:33:05.082319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:05.082289    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:33:05.148732 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:33:05.148698    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:33:20.424886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.424854    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"]
Nov 01 14:33:20.429126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.429106    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.481404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.481363    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"]
Nov 01 14:33:20.481886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.481846    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b435faa8abfb\""
Nov 01 14:33:20.529153 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.529118    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee13fb54a62a\""
Nov 01 14:33:20.529509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.529490    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 14:33:20.529594 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.529490    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 14:33:20.531898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.531884    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vn9m6\""
Nov 01 14:33:20.532133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.532119    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 14:33:20.532169 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.532120    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 14:33:20.588761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.588734    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.588882 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.588847    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-plugins-config-volume\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.588945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.588915    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-routing-overlay-volume\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.588996 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.588948    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dspqf\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-kube-api-access-dspqf\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.690221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.690166    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.690319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.690231    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-plugins-config-volume\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.690319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.690265    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-routing-overlay-volume\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.690319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.690292    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dspqf\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-kube-api-access-dspqf\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.691004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.690983    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-routing-overlay-volume\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.691123 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.691102    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-plugins-config-volume\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.692716 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.692697    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:20.748747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:20.748716    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dspqf\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-kube-api-access-dspqf\") pod \"payload-processing-external-model-6c878cb9c-wxrp2\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") " pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:21.042859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:21.042788    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:21.407291 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:33:21.407257    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b320c5_91db_4ff4_b09a_cff8076bce6b.slice/crio-7583dcda1dee45cd857266cce51dde32cdb1af24b2904f80e89dd2be551d5a9f WatchSource:0}: Error finding container 7583dcda1dee45cd857266cce51dde32cdb1af24b2904f80e89dd2be551d5a9f: Status 404 returned error can't find the container with id 7583dcda1dee45cd857266cce51dde32cdb1af24b2904f80e89dd2be551d5a9f
Nov 01 14:33:21.409770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:21.409749    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"]
Nov 01 14:33:21.575802 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:21.575767    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2" event={"ID":"86b320c5-91db-4ff4-b09a-cff8076bce6b","Type":"ContainerStarted","Data":"8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818"}
Nov 01 14:33:21.576157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:21.575844    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2" event={"ID":"86b320c5-91db-4ff4-b09a-cff8076bce6b","Type":"ContainerStarted","Data":"7583dcda1dee45cd857266cce51dde32cdb1af24b2904f80e89dd2be551d5a9f"}
Nov 01 14:33:21.576157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:21.575901    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:32.324787 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:32.324717    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2" podStartSLOduration=12.324699982 podStartE2EDuration="12.324699982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:21.6903051 +0000 UTC m=+941.855790399" watchObservedRunningTime="2026-11-01 14:33:32.324699982 +0000 UTC m=+952.490185283"
Nov 01 14:33:32.325633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:32.325612    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:33:32.586030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:32.585958    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:33:35.151402 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:33:35.151342    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:33:38.553557 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.553515    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d565598db-9m859"]
Nov 01 14:33:38.557658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.557636    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.575666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.575637    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-plugins-config-volume\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.575781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.575681    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-provider-credentials\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.575828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.575803    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29t9x\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-kube-api-access-29t9x\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.575873 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.575860    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-routing-overlay-volume\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.635677 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.635628    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d565598db-9m859"]
Nov 01 14:33:38.676835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.676797    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29t9x\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-kube-api-access-29t9x\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.677030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.676854    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-routing-overlay-volume\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.677030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.676885    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-plugins-config-volume\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.677030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.676921    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-provider-credentials\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.677663 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.677633    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-routing-overlay-volume\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.677786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.677677    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-plugins-config-volume\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.679667 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.679640    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-provider-credentials\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.733985 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.733954    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29t9x\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-kube-api-access-29t9x\") pod \"payload-processing-external-model-6d565598db-9m859\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") " pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:38.871210 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:38.871180    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:39.239867 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:33:39.239824    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27558173_d96a_451f_aca9_947d9674e5b3.slice/crio-f54a049ca8b14e8281f331d1f4686918d933f0e3a5bf086bf0b5e3911c33eb9f WatchSource:0}: Error finding container f54a049ca8b14e8281f331d1f4686918d933f0e3a5bf086bf0b5e3911c33eb9f: Status 404 returned error can't find the container with id f54a049ca8b14e8281f331d1f4686918d933f0e3a5bf086bf0b5e3911c33eb9f
Nov 01 14:33:39.241851 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:39.241822    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d565598db-9m859"]
Nov 01 14:33:39.697816 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:39.697779    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" event={"ID":"27558173-d96a-451f-aca9-947d9674e5b3","Type":"ContainerStarted","Data":"5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783"}
Nov 01 14:33:39.698196 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:39.697842    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" event={"ID":"27558173-d96a-451f-aca9-947d9674e5b3","Type":"ContainerStarted","Data":"f54a049ca8b14e8281f331d1f4686918d933f0e3a5bf086bf0b5e3911c33eb9f"}
Nov 01 14:33:39.698196 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:39.697962    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:33:39.811613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:39.811561    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" podStartSLOduration=1.811546899 podStartE2EDuration="1.811546899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:39.809803328 +0000 UTC m=+959.975288626" watchObservedRunningTime="2026-11-01 14:33:39.811546899 +0000 UTC m=+959.977032196"
Nov 01 14:33:40.617540 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.617502    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"]
Nov 01 14:33:40.622641 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.622618    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.677482 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.677453    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-00a787-openshift-default-dockercfg-v7d89\""
Nov 01 14:33:40.697031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697001    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqfq\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-kube-api-access-6gqfq\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.697161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697039    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.697161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697065    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.697253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697192    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.697253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697228    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.697328 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697304    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.697328 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697321    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.697418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697338    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.697501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.697482    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f950b9a7-0899-41ea-bea1-820bd450a92c-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.734014 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.733726    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"]
Nov 01 14:33:40.798448 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798416    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.798637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798499    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.798637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798566    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.798637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798612    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.798637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798633    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.798847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798660    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.798847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798731    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f950b9a7-0899-41ea-bea1-820bd450a92c-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.798847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798821    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqfq\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-kube-api-access-6gqfq\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.799032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.798880    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.799089 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.799029    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.799229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.799205    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.799300 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.799284    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.799364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.799346    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.799891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.799866    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f950b9a7-0899-41ea-bea1-820bd450a92c-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.800808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.800778    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.801119 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.801101    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.855624 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.855597    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.855916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.855895    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqfq\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-kube-api-access-6gqfq\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:40.935859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:40.935781    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:41.356255 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:33:41.356215    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf950b9a7_0899_41ea_bea1_820bd450a92c.slice/crio-2ac02046a5ea9b7076a38d1a40e556b2a16539efb6102acf13e7417f5bae85d6 WatchSource:0}: Error finding container 2ac02046a5ea9b7076a38d1a40e556b2a16539efb6102acf13e7417f5bae85d6: Status 404 returned error can't find the container with id 2ac02046a5ea9b7076a38d1a40e556b2a16539efb6102acf13e7417f5bae85d6
Nov 01 14:33:41.358151 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:41.358130    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"]
Nov 01 14:33:41.358838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:41.358802    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:41.358918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:41.358866    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:41.358918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:41.358895    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:41.710013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:41.709930    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" event={"ID":"f950b9a7-0899-41ea-bea1-820bd450a92c","Type":"ContainerStarted","Data":"e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50"}
Nov 01 14:33:41.710013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:41.709988    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" event={"ID":"f950b9a7-0899-41ea-bea1-820bd450a92c","Type":"ContainerStarted","Data":"2ac02046a5ea9b7076a38d1a40e556b2a16539efb6102acf13e7417f5bae85d6"}
Nov 01 14:33:41.836409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:41.831452    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" podStartSLOduration=1.831433048 podStartE2EDuration="1.831433048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:41.830010191 +0000 UTC m=+961.995495875" watchObservedRunningTime="2026-11-01 14:33:41.831433048 +0000 UTC m=+961.996918351"
Nov 01 14:33:41.936881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:41.936851    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:42.937948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:42.937917    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 14:33:42.938332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:42.937968    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 14:33:43.936405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:43.936363    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 14:33:43.936618 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:43.936436    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 14:33:44.936866 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:44.936833    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 14:33:44.937257 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:44.936885    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 14:33:45.936960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:45.936929    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 14:33:45.937357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:45.936982    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 14:33:46.936731 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:46.936683    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 14:33:46.936908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:46.936755    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 14:33:47.974991 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:47.974959    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"]
Nov 01 14:33:47.979621 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:47.979602    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.023882 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.023850    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"]
Nov 01 14:33:48.027610 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.027592    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.075476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.075442    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"]
Nov 01 14:33:48.084769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.084736    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.084914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.084789    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.084914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.084846    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.084914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.084889    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.085084 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.084927    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.085084 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085014    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.085084 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085070    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.085236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085102    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.085236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085144    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.085236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085175    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.085236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085224    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4v9\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-kube-api-access-zk4v9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.085499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085252    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.085499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085276    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.085499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085314    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.085499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085340    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b73d126-74d5-4c66-8e08-ad5a253f509d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.085499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085401    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5hj\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-kube-api-access-gt5hj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.085499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085435    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.085771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.085550    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.137093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.137054    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"]
Nov 01 14:33:48.140186 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.140155    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:48.142246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.142224    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:48.142936 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.142915    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:33:48.186782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.186750    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.186953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.186846    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.186953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.186898    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.186953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.186938    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.186969    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187029    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187081    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187141    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187170    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187215    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187239    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187249    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187320    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4v9\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-kube-api-access-zk4v9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187355    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187399    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187457    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187485    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b73d126-74d5-4c66-8e08-ad5a253f509d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187484    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187518    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5hj\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-kube-api-access-gt5hj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187549    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187625    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187725    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.187980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187722    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187802    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.187980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.187961    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.188320 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.188292    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.188418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.188314    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.189142 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.189115    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b73d126-74d5-4c66-8e08-ad5a253f509d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.190695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.190652    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.191035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.191015    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.191633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.191613    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.192579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.192558    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.245435 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.245346    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.245895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.245869    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5hj\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-kube-api-access-gt5hj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-j2pxz\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.247020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.246996    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4v9\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-kube-api-access-zk4v9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.248428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.248368    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8dh5f\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.293881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.293857    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:48.348781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.345607    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:48.713960 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:33:48.713919    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826dbf37_aeb7_4cfe_9a0f_f98406cbcfd9.slice/crio-a705e38777f000dfd420b301b870337c3427d74e445573b4f1c2435abf6c5ca1 WatchSource:0}: Error finding container a705e38777f000dfd420b301b870337c3427d74e445573b4f1c2435abf6c5ca1: Status 404 returned error can't find the container with id a705e38777f000dfd420b301b870337c3427d74e445573b4f1c2435abf6c5ca1
Nov 01 14:33:48.715862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.715835    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"]
Nov 01 14:33:48.716173 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.716147    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:48.716217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.716203    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:48.716254 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.716231    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:48.755210 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.755186    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" event={"ID":"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9","Type":"ContainerStarted","Data":"a705e38777f000dfd420b301b870337c3427d74e445573b4f1c2435abf6c5ca1"}
Nov 01 14:33:48.763759 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:33:48.763731    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b73d126_74d5_4c66_8e08_ad5a253f509d.slice/crio-9512fecda228bafdeadf85ef54635b345aa3389ffb86a83342feb6294f5bffab WatchSource:0}: Error finding container 9512fecda228bafdeadf85ef54635b345aa3389ffb86a83342feb6294f5bffab: Status 404 returned error can't find the container with id 9512fecda228bafdeadf85ef54635b345aa3389ffb86a83342feb6294f5bffab
Nov 01 14:33:48.765789 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.765768    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"]
Nov 01 14:33:48.766327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.766298    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:48.766369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.766358    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:48.766450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:48.766436    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:33:49.761774 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:49.761564    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" event={"ID":"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9","Type":"ContainerStarted","Data":"ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3"}
Nov 01 14:33:49.763459 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:49.763433    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" event={"ID":"6b73d126-74d5-4c66-8e08-ad5a253f509d","Type":"ContainerStarted","Data":"7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec"}
Nov 01 14:33:49.763535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:49.763477    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" event={"ID":"6b73d126-74d5-4c66-8e08-ad5a253f509d","Type":"ContainerStarted","Data":"9512fecda228bafdeadf85ef54635b345aa3389ffb86a83342feb6294f5bffab"}
Nov 01 14:33:49.930816 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:49.930753    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podStartSLOduration=2.930735684 podStartE2EDuration="2.930735684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:49.929963413 +0000 UTC m=+970.095448715" watchObservedRunningTime="2026-11-01 14:33:49.930735684 +0000 UTC m=+970.096220985"
Nov 01 14:33:50.047778 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:50.047675    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podStartSLOduration=3.047659672 podStartE2EDuration="3.047659672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:50.045863178 +0000 UTC m=+970.211348473" watchObservedRunningTime="2026-11-01 14:33:50.047659672 +0000 UTC m=+970.213144970"
Nov 01 14:33:50.300467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:50.300058    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:50.300467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:50.300173    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-j2pxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 14:33:50.300467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:50.300221    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 14:33:50.348409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:50.346773    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:50.348755 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:50.348731    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8dh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 14:33:50.348978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:50.348950    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 14:33:50.733528 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:50.733485    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" podUID="27558173-d96a-451f-aca9-947d9674e5b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:33:51.299748 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:51.299714    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-j2pxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 14:33:51.300340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:51.300309    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 14:33:51.346997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:51.346966    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8dh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 14:33:51.347223 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:51.347198    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 14:33:52.302416 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:52.298711    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-j2pxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 14:33:52.302416 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:52.298770    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 14:33:52.347854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:52.347816    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8dh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 14:33:52.348114 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:52.348086    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 14:33:53.312411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:53.298720    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-j2pxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 14:33:53.312411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:53.298781    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 14:33:53.346901 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:53.346869    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8dh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 14:33:53.347151 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:53.347129    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 14:33:54.295805 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:54.295763    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-j2pxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 14:33:54.296116 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:54.296082    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 14:33:54.347564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:54.347531    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8dh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 14:33:54.348020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:54.347995    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 14:33:55.294192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:55.294154    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-j2pxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 14:33:55.294364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:55.294220    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 14:33:55.347183 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:55.347144    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8dh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 14:33:55.347369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:55.347210    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 14:33:56.294966 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:56.294932    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-j2pxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 14:33:56.295469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:56.295001    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 14:33:56.347693 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:56.347653    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8dh5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 14:33:56.347857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:56.347736    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 14:33:57.298601 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:57.298568    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:57.299012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:57.298841    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:57.299535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:57.299520    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:33:57.351158 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:57.351132    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:57.351433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:57.351414    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:33:57.352138 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:33:57.352120    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:34:00.704501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:00.704449    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" podUID="27558173-d96a-451f-aca9-947d9674e5b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:34:03.102779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.102744    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"]
Nov 01 14:34:03.106814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.106797    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.196212 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.196185    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"]
Nov 01 14:34:03.266522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.266489    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.266683 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.266542    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvgfv\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-kube-api-access-mvgfv\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.266683 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.266643    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.266792 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.266689    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.266848 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.266789    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.266896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.266851    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.266972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.266904    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.267035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.266987    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.267035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.267015    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368084 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368010    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvgfv\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-kube-api-access-mvgfv\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368084 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368069    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368095    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368118    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368143    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368190    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368217    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368241    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368699 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368553    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368699 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368617    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368777 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368729    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.368811 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.368792    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.369057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.369042    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.369099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.369083    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.370582 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.370559    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.370746 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.370728    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.424527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.424499    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.424639 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.424580    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvgfv\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-kube-api-access-mvgfv\") pod \"maas-default-gateway-openshift-default-7f5667f86c-mhwvk\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:03.721705 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:03.721616    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:04.135865 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:34:04.135832    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780b6486_e5ca_4a6c_a6b6_4b7d5d2def5d.slice/crio-4e4ffed024557b2cf52dec09f77426300802e42cb5c906341916c88548cd2728 WatchSource:0}: Error finding container 4e4ffed024557b2cf52dec09f77426300802e42cb5c906341916c88548cd2728: Status 404 returned error can't find the container with id 4e4ffed024557b2cf52dec09f77426300802e42cb5c906341916c88548cd2728
Nov 01 14:34:04.136989 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:04.136960    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"]
Nov 01 14:34:04.138167 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:04.138131    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:04.138267 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:04.138191    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:04.138267 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:04.138219    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:04.851590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:04.851553    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" event={"ID":"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d","Type":"ContainerStarted","Data":"59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0"}
Nov 01 14:34:04.851762 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:04.851614    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" event={"ID":"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d","Type":"ContainerStarted","Data":"4e4ffed024557b2cf52dec09f77426300802e42cb5c906341916c88548cd2728"}
Nov 01 14:34:05.156890 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:05.156795    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:34:05.186220 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:05.186164    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podStartSLOduration=2.186149936 podStartE2EDuration="2.186149936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:05.072867572 +0000 UTC m=+985.238352871" watchObservedRunningTime="2026-11-01 14:34:05.186149936 +0000 UTC m=+985.351635235"
Nov 01 14:34:05.188141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:05.188117    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:34:05.722685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:05.722648    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:05.724630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:05.724600    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-mhwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 14:34:05.724757 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:05.724659    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 14:34:06.723411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:06.722598    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-mhwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 14:34:06.723411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:06.722664    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 14:34:07.722659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:07.722616    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-mhwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 14:34:07.722830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:07.722699    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 14:34:08.722680 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:08.722649    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-mhwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 14:34:08.723088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:08.722720    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 14:34:09.722617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:09.722579    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-mhwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 14:34:09.722782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:09.722641    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 14:34:10.704609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:10.704572    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" podUID="27558173-d96a-451f-aca9-947d9674e5b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:34:10.722778 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:10.722741    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-mhwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 14:34:10.723242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:10.722788    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 14:34:11.722595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:11.722559    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-mhwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 14:34:11.722775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:11.722625    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 14:34:12.841461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:12.841424    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:12.841881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:12.841824    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:12.842746 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:12.842724    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 14:34:13.961371 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:13.961344    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d565598db-9m859"]
Nov 01 14:34:13.962720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:13.962692    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" podUID="27558173-d96a-451f-aca9-947d9674e5b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:34:13.986978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:13.986948    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"]
Nov 01 14:34:13.992060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:13.992023    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.019687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.019657    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4v69\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-kube-api-access-f4v69\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.019842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.019698    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-provider-credentials\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.019949 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.019929    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-plugins-config-volume\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.020126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.020102    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-routing-overlay-volume\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.074755 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.074722    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"]
Nov 01 14:34:14.121405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.121348    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-provider-credentials\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.121569 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.121470    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-plugins-config-volume\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.121569 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.121556    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-routing-overlay-volume\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.121725 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.121657    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4v69\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-kube-api-access-f4v69\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.122207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.122184    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-routing-overlay-volume\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.122302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.122230    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-plugins-config-volume\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.123781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.123761    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-provider-credentials\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.178742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.178704    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4v69\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-kube-api-access-f4v69\") pod \"payload-processing-external-model-5dfd5db6bc-lflcl\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.305836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.305746    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:14.673025 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:34:14.672986    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30248ad2_327c_4ce8_ab80_da11311cd301.slice/crio-98213604f5b44d17ce3c42fc31403995674b5546c434a8cb43eec5ec1e1b6923 WatchSource:0}: Error finding container 98213604f5b44d17ce3c42fc31403995674b5546c434a8cb43eec5ec1e1b6923: Status 404 returned error can't find the container with id 98213604f5b44d17ce3c42fc31403995674b5546c434a8cb43eec5ec1e1b6923
Nov 01 14:34:14.674009 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.673981    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"]
Nov 01 14:34:14.913563 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.913533    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" event={"ID":"30248ad2-327c-4ce8-ab80-da11311cd301","Type":"ContainerStarted","Data":"798fd4222f56b447909dc50defe09c3e7090753f34892740b53c6443cc9cdecc"}
Nov 01 14:34:14.913735 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.913581    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" event={"ID":"30248ad2-327c-4ce8-ab80-da11311cd301","Type":"ContainerStarted","Data":"98213604f5b44d17ce3c42fc31403995674b5546c434a8cb43eec5ec1e1b6923"}
Nov 01 14:34:14.913735 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:14.913623    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:34:15.028108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:15.028016    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" podStartSLOduration=2.028003656 podStartE2EDuration="2.028003656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:15.027474673 +0000 UTC m=+995.192959971" watchObservedRunningTime="2026-11-01 14:34:15.028003656 +0000 UTC m=+995.193488953"
Nov 01 14:34:18.965905 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:18.965831    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" podUID="27558173-d96a-451f-aca9-947d9674e5b3" containerName="payload-processing" containerID="cri-o://5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783" gracePeriod=25
Nov 01 14:34:19.420074 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.419995    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:34:19.504261 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.504214    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-plugins-config-volume\") pod \"27558173-d96a-451f-aca9-947d9674e5b3\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") "
Nov 01 14:34:19.504482 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.504360    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-kube-api-access-29t9x\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-kube-api-access-29t9x\") pod \"27558173-d96a-451f-aca9-947d9674e5b3\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") "
Nov 01 14:34:19.504563 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.504500    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-provider-credentials\") pod \"27558173-d96a-451f-aca9-947d9674e5b3\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") "
Nov 01 14:34:19.504619 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.504565    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-routing-overlay-volume\") pod \"27558173-d96a-451f-aca9-947d9674e5b3\" (UID: \"27558173-d96a-451f-aca9-947d9674e5b3\") "
Nov 01 14:34:19.505474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.505443    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-routing-overlay-volume" pod "27558173-d96a-451f-aca9-947d9674e5b3" (UID: "27558173-d96a-451f-aca9-947d9674e5b3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:34:19.505986 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.505962    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-plugins-config-volume" pod "27558173-d96a-451f-aca9-947d9674e5b3" (UID: "27558173-d96a-451f-aca9-947d9674e5b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:34:19.508993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.508970    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-kube-api-access-29t9x" pod "27558173-d96a-451f-aca9-947d9674e5b3" (UID: "27558173-d96a-451f-aca9-947d9674e5b3"). InnerVolumeSpecName "kube-api-access-29t9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:34:19.510574 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.510548    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-provider-credentials" pod "27558173-d96a-451f-aca9-947d9674e5b3" (UID: "27558173-d96a-451f-aca9-947d9674e5b3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:34:19.617471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.616552    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:34:19.617471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.616593    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:34:19.617471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.616609    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27558173-d96a-451f-aca9-947d9674e5b3-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:34:19.617471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.616624    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29t9x\" (UniqueName: \"kubernetes.io/projected/27558173-d96a-451f-aca9-947d9674e5b3-kube-api-access-29t9x\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:34:19.911460 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.910726    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"]
Nov 01 14:34:19.911634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.911518    5423 cpu_manager.go:414] "containerMap: removing container" podUID="27558173-d96a-451f-aca9-947d9674e5b3" containerName="payload-processing"
Nov 01 14:34:19.911634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.911539    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27558173-d96a-451f-aca9-947d9674e5b3" containerName="payload-processing"
Nov 01 14:34:19.911759 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.911702    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="27558173-d96a-451f-aca9-947d9674e5b3" containerName="payload-processing"
Nov 01 14:34:19.919948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:19.919923    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.050189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.050152    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.061574 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.061533    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdkkc\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-kube-api-access-jdkkc\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.061981 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.061865    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.062270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.062248    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.062640 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.062618    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.062830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.062803    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.063336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.063214    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.069541 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.069517    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.069700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.069680    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.103412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.092910    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"]
Nov 01 14:34:20.111984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.104696    5423 generic.go:356] "Generic (PLEG): container finished" podID="27558173-d96a-451f-aca9-947d9674e5b3" containerID="5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783" exitCode=0
Nov 01 14:34:20.123534 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.104765    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.313212    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" event={"ID":"27558173-d96a-451f-aca9-947d9674e5b3","Type":"ContainerDied","Data":"5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783"}
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.313294    5423 scope.go:122] "RemoveContainer" containerID="5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.314883    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.314961    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.315002    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.315074    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.315129    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.315178    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.315253    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.315303    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdkkc\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-kube-api-access-jdkkc\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.315353    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.316323    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.316701    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.316988    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.338410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.326763    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.371445 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.340748    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d565598db-9m859" event={"ID":"27558173-d96a-451f-aca9-947d9674e5b3","Type":"ContainerDied","Data":"f54a049ca8b14e8281f331d1f4686918d933f0e3a5bf086bf0b5e3911c33eb9f"}
Nov 01 14:34:20.544410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.538570    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.614201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.614127    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.665411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.626591    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.665411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.646276    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdkkc\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-kube-api-access-jdkkc\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.681280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.680930    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.762860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.762832    5423 scope.go:122] "RemoveContainer" containerID="5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783"
Nov 01 14:34:20.767704 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:20.767673    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783\": container with ID starting with 5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783 not found: ID does not exist" containerID="5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783"
Nov 01 14:34:20.767883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.767858    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783"} err="failed to get container status \"5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783\": rpc error: code = NotFound desc = could not find container \"5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783\": container with ID starting with 5cb251bc7c6c68c4a395566e47492aad0a81f4e931a3788ea016a4be3d3ec783 not found: ID does not exist"
Nov 01 14:34:20.834410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.832434    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d565598db-9m859"]
Nov 01 14:34:20.847603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.847573    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:20.848948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:20.848376    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d565598db-9m859"]
Nov 01 14:34:21.783052 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:34:21.783018    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e3aa7b_02db_4b4d_aa3f_d9c74d47f4db.slice/crio-cecbf2fc1639dcc1d104e1c9437d6e0eb4d29a77323eebc6440a498604bfa5fd WatchSource:0}: Error finding container cecbf2fc1639dcc1d104e1c9437d6e0eb4d29a77323eebc6440a498604bfa5fd: Status 404 returned error can't find the container with id cecbf2fc1639dcc1d104e1c9437d6e0eb4d29a77323eebc6440a498604bfa5fd
Nov 01 14:34:21.795888 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:21.793514    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"]
Nov 01 14:34:21.809998 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:21.809954    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:21.810624 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:21.810597    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:21.810787 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:21.810766    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:21.977087 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:21.976636    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27558173-d96a-451f-aca9-947d9674e5b3" path="/var/lib/kubelet/pods/27558173-d96a-451f-aca9-947d9674e5b3/volumes"
Nov 01 14:34:22.194926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:22.194873    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" event={"ID":"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db","Type":"ContainerStarted","Data":"86a05188262021468e1ad4989ca890a44204611bd59cf058edca4df20e1f4a4c"}
Nov 01 14:34:22.195101 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:22.194955    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" event={"ID":"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db","Type":"ContainerStarted","Data":"cecbf2fc1639dcc1d104e1c9437d6e0eb4d29a77323eebc6440a498604bfa5fd"}
Nov 01 14:34:22.311854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:22.311807    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podStartSLOduration=3.311794671 podStartE2EDuration="3.311794671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:22.310855917 +0000 UTC m=+1002.476341222" watchObservedRunningTime="2026-11-01 14:34:22.311794671 +0000 UTC m=+1002.477279970"
Nov 01 14:34:22.849211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:22.849166    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:22.850633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:22.850606    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 14:34:22.850783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:22.850658    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 14:34:23.848324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:23.848293    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 14:34:23.848501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:23.848345    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 14:34:24.849309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:24.849272    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 14:34:24.849745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:24.849330    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 14:34:25.848344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:25.848311    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 14:34:25.848543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:25.848362    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 14:34:25.922486 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:25.922454    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:34:26.848317 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:26.848281    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 14:34:26.848518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:26.848330    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 14:34:27.848667 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:27.848632    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 01 14:34:27.849066 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:27.848686    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 01 14:34:29.029301 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:29.029267    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:29.030432 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:29.030378    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:29.031241 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:29.031224    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:34:32.769647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:32.769612    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:34:35.149297 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:35.149260    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:34:35.920153 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:35.920112    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:34:41.088978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.088944    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"]
Nov 01 14:34:41.094035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.094012    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.181292 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.181263    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"]
Nov 01 14:34:41.237765 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.237731    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86908c75-744e-4256-9377-243eaf12e5c2-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.237923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.237858    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86908c75-744e-4256-9377-243eaf12e5c2-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.237923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.237911    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.237993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.237927    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.238102 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.238078    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.238143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.238129    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89llq\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-kube-api-access-89llq\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.238177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.238165    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.238261 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.238245    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.238324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.238305    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.339732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.339645    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89llq\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-kube-api-access-89llq\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.339732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.339700    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.339958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.339748    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.339958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.339780    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.339958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.339851    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86908c75-744e-4256-9377-243eaf12e5c2-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.339958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.339935    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86908c75-744e-4256-9377-243eaf12e5c2-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.340177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.340003    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.340177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.340027    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.340177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.340071    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.340177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.340107    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.340177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.340139    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.340465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.340375    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.340596 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.340576    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.340772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.340748    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86908c75-744e-4256-9377-243eaf12e5c2-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.342469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.342440    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.342579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.342510    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86908c75-744e-4256-9377-243eaf12e5c2-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.396907 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.396878    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89llq\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-kube-api-access-89llq\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.397036 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.396910    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.408001 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.407982    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:41.822514 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:34:41.822476    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86908c75_744e_4256_9377_243eaf12e5c2.slice/crio-0f74f43ecaab9d8dfc9196863d440c5b4f2a8fd54f1320bd5602d72ea748c23a WatchSource:0}: Error finding container 0f74f43ecaab9d8dfc9196863d440c5b4f2a8fd54f1320bd5602d72ea748c23a: Status 404 returned error can't find the container with id 0f74f43ecaab9d8dfc9196863d440c5b4f2a8fd54f1320bd5602d72ea748c23a
Nov 01 14:34:41.824399 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.824364    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"]
Nov 01 14:34:41.824991 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.824959    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:41.825096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.825045    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:41.825096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:41.825086    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:34:42.363542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:42.363509    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" event={"ID":"86908c75-744e-4256-9377-243eaf12e5c2","Type":"ContainerStarted","Data":"d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec"}
Nov 01 14:34:42.363905 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:42.363568    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" event={"ID":"86908c75-744e-4256-9377-243eaf12e5c2","Type":"ContainerStarted","Data":"0f74f43ecaab9d8dfc9196863d440c5b4f2a8fd54f1320bd5602d72ea748c23a"}
Nov 01 14:34:42.418407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:42.417741    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:42.420303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:42.420271    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 01 14:34:42.420517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:42.420344    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 01 14:34:42.485695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:42.485642    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podStartSLOduration=1.485628452 podStartE2EDuration="1.485628452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:42.484169447 +0000 UTC m=+1022.649654745" watchObservedRunningTime="2026-11-01 14:34:42.485628452 +0000 UTC m=+1022.651113751"
Nov 01 14:34:43.411830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:43.411787    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 01 14:34:43.412492 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:43.412448    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 01 14:34:44.413518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:44.413481    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 01 14:34:44.414100 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:44.414063    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 01 14:34:45.416408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:45.415741    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 01 14:34:45.416408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:45.415813    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 01 14:34:45.922067 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:45.922025    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:34:46.419540 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:46.419501    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 01 14:34:46.419941 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:46.419556    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 01 14:34:47.416411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:47.412773    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 01 14:34:47.416411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:47.412835    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 01 14:34:48.408295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:48.408258    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 01 14:34:48.408690 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:48.408307    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 01 14:34:49.413580 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:49.413550    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:49.413997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:49.413930    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:49.414576 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:49.414558    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:34:54.014562 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:54.014527    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b435faa8abfb: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:54.014562 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:54.014558    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:54.015153 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:54.014634    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials podName:86b320c5-91db-4ff4-b09a-cff8076bce6b nodeName:}" failed. No retries permitted until 2026-11-01 14:34:54.51461428 +0000 UTC m=+1034.680099575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials") pod "payload-processing-external-model-6c878cb9c-wxrp2" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b") : secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:54.520203 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:54.520170    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b435faa8abfb: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:54.520203 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:54.520199    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:54.520438 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:54.520248    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials podName:86b320c5-91db-4ff4-b09a-cff8076bce6b nodeName:}" failed. No retries permitted until 2026-11-01 14:34:55.520233487 +0000 UTC m=+1035.685718762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials") pod "payload-processing-external-model-6c878cb9c-wxrp2" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b") : secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:55.531403 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:55.531362    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b435faa8abfb: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:55.531779 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:55.531410    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:55.531779 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:55.531473    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials podName:86b320c5-91db-4ff4-b09a-cff8076bce6b nodeName:}" failed. No retries permitted until 2026-11-01 14:34:57.531455982 +0000 UTC m=+1037.696941275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials") pod "payload-processing-external-model-6c878cb9c-wxrp2" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b") : secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:55.920601 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:55.920556    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:34:57.549947 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:57.549918    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b435faa8abfb: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:57.549947 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:57.549947    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:57.552314 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:34:57.550006    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials podName:86b320c5-91db-4ff4-b09a-cff8076bce6b nodeName:}" failed. No retries permitted until 2026-11-01 14:35:01.549988792 +0000 UTC m=+1041.715474071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials") pod "payload-processing-external-model-6c878cb9c-wxrp2" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b") : secret "agc-cred-b435faa8abfb" not found
Nov 01 14:34:58.490268 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:34:58.490234    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:35:01.594185 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:01.594154    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b435faa8abfb: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:35:01.594185 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:01.594184    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:35:01.594625 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:01.594235    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials podName:86b320c5-91db-4ff4-b09a-cff8076bce6b nodeName:}" failed. No retries permitted until 2026-11-01 14:35:09.594221338 +0000 UTC m=+1049.759706639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials") pod "payload-processing-external-model-6c878cb9c-wxrp2" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b") : secret "agc-cred-b435faa8abfb" not found
Nov 01 14:35:05.148280 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:05.148241    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:35:05.920965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:05.920932    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:35:07.248540 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:07.248509    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"]
Nov 01 14:35:07.249869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:07.249843    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:35:09.614493 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:09.614459    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b435faa8abfb: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:35:09.614493 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:09.614490    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2: secret "agc-cred-b435faa8abfb" not found
Nov 01 14:35:09.615000 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:09.614562    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials podName:86b320c5-91db-4ff4-b09a-cff8076bce6b nodeName:}" failed. No retries permitted until 2026-11-01 14:35:25.614542135 +0000 UTC m=+1065.780027412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials") pod "payload-processing-external-model-6c878cb9c-wxrp2" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b") : secret "agc-cred-b435faa8abfb" not found
Nov 01 14:35:09.672051 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.672018    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"]
Nov 01 14:35:09.675729 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.675710    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.729135 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.729100    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"]
Nov 01 14:35:09.775527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.775496    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"distinct-2-w3-5b0dcc-kserve-self-signed-certs\""
Nov 01 14:35:09.851428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.821162    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-home\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.851428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.851325    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19a5189-f91f-4561-9fa5-082efe591e41-tls-certs\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.851428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.851407    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-dshm\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.851726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.851529    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xsk\" (UniqueName: \"kubernetes.io/projected/e19a5189-f91f-4561-9fa5-082efe591e41-kube-api-access-j4xsk\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.851726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.851701    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-model-cache\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.851836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.851729    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-tmp-dir\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.953265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.953181    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-home\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.953265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.953216    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19a5189-f91f-4561-9fa5-082efe591e41-tls-certs\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.953265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.953238    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-dshm\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.953609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.953297    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xsk\" (UniqueName: \"kubernetes.io/projected/e19a5189-f91f-4561-9fa5-082efe591e41-kube-api-access-j4xsk\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.953609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.953476    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-model-cache\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.953609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.953506    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-tmp-dir\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.953773 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.953739    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-home\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.954014 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.953990    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-model-cache\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.954236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.954205    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-tmp-dir\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.955646 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.955616    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-dshm\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:09.956334 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:09.956295    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19a5189-f91f-4561-9fa5-082efe591e41-tls-certs\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:10.022363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:10.022336    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xsk\" (UniqueName: \"kubernetes.io/projected/e19a5189-f91f-4561-9fa5-082efe591e41-kube-api-access-j4xsk\") pod \"distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:10.288657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:10.288576    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:10.654835 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:35:10.654797    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode19a5189_f91f_4561_9fa5_082efe591e41.slice/crio-625c65e8f491eaf4812018609e1748d6ea433509452d318e221aa9cdde3f4b97 WatchSource:0}: Error finding container 625c65e8f491eaf4812018609e1748d6ea433509452d318e221aa9cdde3f4b97: Status 404 returned error can't find the container with id 625c65e8f491eaf4812018609e1748d6ea433509452d318e221aa9cdde3f4b97
Nov 01 14:35:10.655367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:10.655036    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"]
Nov 01 14:35:11.563844 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:11.563798    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" event={"ID":"e19a5189-f91f-4561-9fa5-082efe591e41","Type":"ContainerStarted","Data":"d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8"}
Nov 01 14:35:11.564098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:11.563854    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" event={"ID":"e19a5189-f91f-4561-9fa5-082efe591e41","Type":"ContainerStarted","Data":"625c65e8f491eaf4812018609e1748d6ea433509452d318e221aa9cdde3f4b97"}
Nov 01 14:35:11.678800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:11.678743    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podStartSLOduration=2.6787293009999997 podStartE2EDuration="2.678729301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:35:11.678248344 +0000 UTC m=+1051.843733642" watchObservedRunningTime="2026-11-01 14:35:11.678729301 +0000 UTC m=+1051.844214593"
Nov 01 14:35:12.256407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.250702    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing" containerID="cri-o://798fd4222f56b447909dc50defe09c3e7090753f34892740b53c6443cc9cdecc" gracePeriod=25
Nov 01 14:35:12.291450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.289469    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:12.574876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.574781    5423 generic.go:356] "Generic (PLEG): container finished" podID="30248ad2-327c-4ce8-ab80-da11311cd301" containerID="798fd4222f56b447909dc50defe09c3e7090753f34892740b53c6443cc9cdecc" exitCode=0
Nov 01 14:35:12.576301 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.576271    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" event={"ID":"30248ad2-327c-4ce8-ab80-da11311cd301","Type":"ContainerDied","Data":"798fd4222f56b447909dc50defe09c3e7090753f34892740b53c6443cc9cdecc"}
Nov 01 14:35:12.576429 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.576332    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl" event={"ID":"30248ad2-327c-4ce8-ab80-da11311cd301","Type":"ContainerDied","Data":"98213604f5b44d17ce3c42fc31403995674b5546c434a8cb43eec5ec1e1b6923"}
Nov 01 14:35:12.576429 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.576360    5423 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98213604f5b44d17ce3c42fc31403995674b5546c434a8cb43eec5ec1e1b6923"
Nov 01 14:35:12.605818 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.605789    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:35:12.689466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.689419    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-routing-overlay-volume\") pod \"30248ad2-327c-4ce8-ab80-da11311cd301\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") "
Nov 01 14:35:12.689957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.689544    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-kube-api-access-f4v69\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-kube-api-access-f4v69\") pod \"30248ad2-327c-4ce8-ab80-da11311cd301\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") "
Nov 01 14:35:12.689957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.689674    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-provider-credentials\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-provider-credentials\") pod \"30248ad2-327c-4ce8-ab80-da11311cd301\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") "
Nov 01 14:35:12.689957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.689739    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-plugins-config-volume\") pod \"30248ad2-327c-4ce8-ab80-da11311cd301\" (UID: \"30248ad2-327c-4ce8-ab80-da11311cd301\") "
Nov 01 14:35:12.689957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.689841    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-routing-overlay-volume" pod "30248ad2-327c-4ce8-ab80-da11311cd301" (UID: "30248ad2-327c-4ce8-ab80-da11311cd301"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:35:12.690182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.690128    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-plugins-config-volume" pod "30248ad2-327c-4ce8-ab80-da11311cd301" (UID: "30248ad2-327c-4ce8-ab80-da11311cd301"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:35:12.690243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.690180    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:35:12.693345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.693318    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-kube-api-access-f4v69" pod "30248ad2-327c-4ce8-ab80-da11311cd301" (UID: "30248ad2-327c-4ce8-ab80-da11311cd301"). InnerVolumeSpecName "kube-api-access-f4v69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:35:12.709416 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.707530    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-provider-credentials" pod "30248ad2-327c-4ce8-ab80-da11311cd301" (UID: "30248ad2-327c-4ce8-ab80-da11311cd301"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:35:12.792408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.791401    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30248ad2-327c-4ce8-ab80-da11311cd301-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:35:12.792408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.791441    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4v69\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-kube-api-access-f4v69\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:35:12.792408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:12.791458    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30248ad2-327c-4ce8-ab80-da11311cd301-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:35:13.601409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:13.588373    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"
Nov 01 14:35:13.781177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:13.781141    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"]
Nov 01 14:35:13.853908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:13.853823    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-lflcl"]
Nov 01 14:35:13.974325 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:13.974293    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" path="/var/lib/kubelet/pods/30248ad2-327c-4ce8-ab80-da11311cd301/volumes"
Nov 01 14:35:19.630532 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:19.630493    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"]
Nov 01 14:35:20.288850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:20.288816    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:20.301430 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:20.301408    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:20.678048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:20.678022    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:35:24.630898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:24.630851    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2" podUID="86b320c5-91db-4ff4-b09a-cff8076bce6b" containerName="payload-processing" containerID="cri-o://8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818" gracePeriod=25
Nov 01 14:35:24.886800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:24.886740    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:35:24.999604 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:24.999572    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-plugins-config-volume\") pod \"86b320c5-91db-4ff4-b09a-cff8076bce6b\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") "
Nov 01 14:35:24.999798 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:24.999616    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-kube-api-access-dspqf\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-kube-api-access-dspqf\") pod \"86b320c5-91db-4ff4-b09a-cff8076bce6b\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") "
Nov 01 14:35:24.999798 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:24.999717    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials\") pod \"86b320c5-91db-4ff4-b09a-cff8076bce6b\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") "
Nov 01 14:35:24.999798 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:24.999778    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-routing-overlay-volume\") pod \"86b320c5-91db-4ff4-b09a-cff8076bce6b\" (UID: \"86b320c5-91db-4ff4-b09a-cff8076bce6b\") "
Nov 01 14:35:25.000223 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.000190    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-routing-overlay-volume" pod "86b320c5-91db-4ff4-b09a-cff8076bce6b" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:35:25.000361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.000235    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-plugins-config-volume" pod "86b320c5-91db-4ff4-b09a-cff8076bce6b" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:35:25.001940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.001916    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-kube-api-access-dspqf" pod "86b320c5-91db-4ff4-b09a-cff8076bce6b" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b"). InnerVolumeSpecName "kube-api-access-dspqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:35:25.012404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.002579    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials" pod "86b320c5-91db-4ff4-b09a-cff8076bce6b" (UID: "86b320c5-91db-4ff4-b09a-cff8076bce6b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:35:25.101041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.101008    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:35:25.101041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.101035    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:35:25.101041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.101047    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b320c5-91db-4ff4-b09a-cff8076bce6b-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:35:25.101262 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.101056    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dspqf\" (UniqueName: \"kubernetes.io/projected/86b320c5-91db-4ff4-b09a-cff8076bce6b-kube-api-access-dspqf\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:35:25.693507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.693475    5423 generic.go:356] "Generic (PLEG): container finished" podID="86b320c5-91db-4ff4-b09a-cff8076bce6b" containerID="8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818" exitCode=0
Nov 01 14:35:25.693993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.693535    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"
Nov 01 14:35:25.693993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.693561    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2" event={"ID":"86b320c5-91db-4ff4-b09a-cff8076bce6b","Type":"ContainerDied","Data":"8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818"}
Nov 01 14:35:25.693993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.693654    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2" event={"ID":"86b320c5-91db-4ff4-b09a-cff8076bce6b","Type":"ContainerDied","Data":"7583dcda1dee45cd857266cce51dde32cdb1af24b2904f80e89dd2be551d5a9f"}
Nov 01 14:35:25.693993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.693689    5423 scope.go:122] "RemoveContainer" containerID="8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818"
Nov 01 14:35:25.704077 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.704062    5423 scope.go:122] "RemoveContainer" containerID="8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818"
Nov 01 14:35:25.704315 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:25.704296    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818\": container with ID starting with 8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818 not found: ID does not exist" containerID="8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818"
Nov 01 14:35:25.704374 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.704328    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818"} err="failed to get container status \"8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818\": rpc error: code = NotFound desc = could not find container \"8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818\": container with ID starting with 8f946f2c0bd27e9ceeb0c16b73db996a14b6b0f9c3b6e48be8653aa2a4dab818 not found: ID does not exist"
Nov 01 14:35:25.856456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.856422    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"]
Nov 01 14:35:25.906221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.906192    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c878cb9c-wxrp2"]
Nov 01 14:35:25.959983 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:25.959911    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b320c5-91db-4ff4-b09a-cff8076bce6b" path="/var/lib/kubelet/pods/86b320c5-91db-4ff4-b09a-cff8076bce6b/volumes"
Nov 01 14:35:28.219465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:28.219433    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:35:30.834226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:30.834176    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 01 14:35:31.305481 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:31.295495    5423 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:35:31.673541 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:31.673488    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:35:31.835534 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:31.835481    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:35:31.836760 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:31.836736    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:35:32.390411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:32.370872    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:35:35.149610 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:35:35.149570    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:35:46.247998 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.247964    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"]
Nov 01 14:35:46.248683 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.248667    5423 cpu_manager.go:414] "containerMap: removing container" podUID="86b320c5-91db-4ff4-b09a-cff8076bce6b" containerName="payload-processing"
Nov 01 14:35:46.248733 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.248686    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b320c5-91db-4ff4-b09a-cff8076bce6b" containerName="payload-processing"
Nov 01 14:35:46.248770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.248732    5423 cpu_manager.go:414] "containerMap: removing container" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing"
Nov 01 14:35:46.248770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.248740    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing"
Nov 01 14:35:46.248845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.248834    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="30248ad2-327c-4ce8-ab80-da11311cd301" containerName="payload-processing"
Nov 01 14:35:46.248879 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.248853    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b320c5-91db-4ff4-b09a-cff8076bce6b" containerName="payload-processing"
Nov 01 14:35:46.254006 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.253987    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.303424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.303369    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"]
Nov 01 14:35:46.351617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.351591    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"distinct-w2-007270-kserve-self-signed-certs\""
Nov 01 14:35:46.351617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.351594    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"kube-root-ca.crt\""
Nov 01 14:35:46.356158 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.356139    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"openshift-service-ca.crt\""
Nov 01 14:35:46.356274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.356165    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"default-dockercfg-9t7qz\""
Nov 01 14:35:46.368745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.368714    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e30a1b-095a-450b-98af-ecae621e1047-tls-certs\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.368869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.368800    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-home\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.368869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.368853    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-tmp-dir\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.368969 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.368910    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-dshm\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.369018 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.369005    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-model-cache\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.369058 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.369037    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4drw\" (UniqueName: \"kubernetes.io/projected/85e30a1b-095a-450b-98af-ecae621e1047-kube-api-access-n4drw\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.469984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.469946    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e30a1b-095a-450b-98af-ecae621e1047-tls-certs\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.470159 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.470002    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-home\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.470159 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.470026    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-tmp-dir\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.470159 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.470074    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-dshm\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.470159 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.470110    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-model-cache\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.470159 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.470140    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4drw\" (UniqueName: \"kubernetes.io/projected/85e30a1b-095a-450b-98af-ecae621e1047-kube-api-access-n4drw\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.470544 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.470513    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-home\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.470667 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.470604    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-tmp-dir\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.470667 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.470622    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-model-cache\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.472327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.472303    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-dshm\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.472607 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.472589    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e30a1b-095a-450b-98af-ecae621e1047-tls-certs\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.527293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.527230    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4drw\" (UniqueName: \"kubernetes.io/projected/85e30a1b-095a-450b-98af-ecae621e1047-kube-api-access-n4drw\") pod \"distinct-w2-007270-kserve-84bfb5d9c-4l4c9\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") " pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:46.566737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:46.566710    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:47.026206 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:35:47.026168    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85e30a1b_095a_450b_98af_ecae621e1047.slice/crio-5ec13050b0e208467c0b3b103ae94991a3ef93f95134ac64982b233971f59e65 WatchSource:0}: Error finding container 5ec13050b0e208467c0b3b103ae94991a3ef93f95134ac64982b233971f59e65: Status 404 returned error can't find the container with id 5ec13050b0e208467c0b3b103ae94991a3ef93f95134ac64982b233971f59e65
Nov 01 14:35:47.028688 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:47.028664    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"]
Nov 01 14:35:47.934970 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:47.934926    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" event={"ID":"85e30a1b-095a-450b-98af-ecae621e1047","Type":"ContainerStarted","Data":"2ed59a259d2cf240cd512140e3ad0090027d4b3a3be230d321aa24358680bc66"}
Nov 01 14:35:47.935359 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:47.934984    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" event={"ID":"85e30a1b-095a-450b-98af-ecae621e1047","Type":"ContainerStarted","Data":"5ec13050b0e208467c0b3b103ae94991a3ef93f95134ac64982b233971f59e65"}
Nov 01 14:35:48.050184 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:48.050134    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" podStartSLOduration=2.050121533 podStartE2EDuration="2.050121533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:35:48.049300449 +0000 UTC m=+1088.214785746" watchObservedRunningTime="2026-11-01 14:35:48.050121533 +0000 UTC m=+1088.215606830"
Nov 01 14:35:48.567799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:48.567771    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:56.567912 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.567817    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:56.580739 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.580707    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:56.685272 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.685240    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"]
Nov 01 14:35:56.689264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.689241    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.789846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.789809    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.789846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.789845    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.790066 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.789874    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.790110 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.790076    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.790145 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.790128    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.790177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.790169    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.790214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.790191    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrtf2\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-kube-api-access-qrtf2\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.790214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.790209    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.790279 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.790229    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.798761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.798730    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"]
Nov 01 14:35:56.891517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891442    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.891517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891502    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.891742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891520    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.891742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891542    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrtf2\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-kube-api-access-qrtf2\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.891742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891559    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.891742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891577    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.891742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891645    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.891742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891671    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.891742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891702    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.892076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891929    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.892076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.891981    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.892216 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.892193    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.892278 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.892234    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.892431 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.892403    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.894214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.894186    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.894456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.894439    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.948966 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.948933    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.949157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.949136    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrtf2\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-kube-api-access-qrtf2\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:56.999732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:56.999705    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:35:57.003004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:57.002987    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:57.418618 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:35:57.418591    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8fa78c_ccfb_4655_8245_2b7e74bfda44.slice/crio-d457b317ec28f9eeb22f2ea50692122089532c42af017515f5ef2c4f70230aa5 WatchSource:0}: Error finding container d457b317ec28f9eeb22f2ea50692122089532c42af017515f5ef2c4f70230aa5: Status 404 returned error can't find the container with id d457b317ec28f9eeb22f2ea50692122089532c42af017515f5ef2c4f70230aa5
Nov 01 14:35:57.420850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:57.420826    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"]
Nov 01 14:35:57.420947 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:57.420896    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:35:57.420982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:57.420967    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:35:57.421016 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:57.421005    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:35:57.860027 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:57.859992    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:35:58.003950 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:58.003915    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" event={"ID":"fa8fa78c-ccfb-4655-8245-2b7e74bfda44","Type":"ContainerStarted","Data":"2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712"}
Nov 01 14:35:58.004238 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:58.004216    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" event={"ID":"fa8fa78c-ccfb-4655-8245-2b7e74bfda44","Type":"ContainerStarted","Data":"d457b317ec28f9eeb22f2ea50692122089532c42af017515f5ef2c4f70230aa5"}
Nov 01 14:35:58.006168 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:58.006147    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:35:58.006427 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:58.006409    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:35:58.006576 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:58.006555    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 14:35:59.011661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:59.011617    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:35:59.012171 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:35:59.011682    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 14:36:00.007411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:00.007192    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:36:00.007411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:00.007268    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 14:36:01.013414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:01.010475    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:36:01.013414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:01.010535    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 14:36:02.010756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:02.010719    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:36:02.011070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:02.011037    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 14:36:03.014410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:03.011730    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:36:03.014410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:03.011790    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 14:36:04.003718 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:04.003678    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:36:04.003938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:04.003733    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 14:36:05.008023 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:05.007986    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:36:05.008445 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:05.008268    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:36:05.008940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:05.008925    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 14:36:05.150064 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:36:05.150031    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:36:05.226617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:05.226559    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podStartSLOduration=9.226545448 podStartE2EDuration="9.226545448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:35:58.124741154 +0000 UTC m=+1098.290226458" watchObservedRunningTime="2026-11-01 14:36:05.226545448 +0000 UTC m=+1105.392030797"
Nov 01 14:36:21.066743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:21.066706    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:36:35.234121 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:36:35.234068    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:36:35.988807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:35.988753    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"]
Nov 01 14:36:35.995758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:35.994988    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.100874 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.100840    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"]
Nov 01 14:36:36.101099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101076    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.101185 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101110    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-kube-api-access-rhhmn\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.101185 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101150    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.101303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101201    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.101303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101260    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.101303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101298    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.101509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101331    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.101509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101435    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.101632 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.101549    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203096    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203147    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203192    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203303    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203362    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203432    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-kube-api-access-rhhmn\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203483    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203521    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203681    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203757 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203715    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203812 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203795    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.203951 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203930    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.204017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.203965    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.204087 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.204064    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.205983 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.205963    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.206091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.206008    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.261163 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.261100    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.261761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.261739    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-kube-api-access-rhhmn\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.312356 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.312328    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:36.728101 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.728070    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"]
Nov 01 14:36:36.761630 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:36:36.761603    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d9712d_03fe_41d7_b79c_f5f0de92c8a1.slice/crio-8a40ee809d515d310de06c58a75ee66309b68e23f6faaf5bff78702b051d1284 WatchSource:0}: Error finding container 8a40ee809d515d310de06c58a75ee66309b68e23f6faaf5bff78702b051d1284: Status 404 returned error can't find the container with id 8a40ee809d515d310de06c58a75ee66309b68e23f6faaf5bff78702b051d1284
Nov 01 14:36:36.763964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.763931    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:36:36.764052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.764003    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:36:36.764052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:36.764038    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:36:37.326297 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:37.326259    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" event={"ID":"98d9712d-03fe-41d7-b79c-f5f0de92c8a1","Type":"ContainerStarted","Data":"74c7bff8abeda310c069eecc62c49fd86f3f9a307be1400a8bc78a63a1bb8dbc"}
Nov 01 14:36:37.326842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:37.326322    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" event={"ID":"98d9712d-03fe-41d7-b79c-f5f0de92c8a1","Type":"ContainerStarted","Data":"8a40ee809d515d310de06c58a75ee66309b68e23f6faaf5bff78702b051d1284"}
Nov 01 14:36:37.444396 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:37.444329    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podStartSLOduration=2.444314027 podStartE2EDuration="2.444314027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:37.443052959 +0000 UTC m=+1137.608538256" watchObservedRunningTime="2026-11-01 14:36:37.444314027 +0000 UTC m=+1137.609799325"
Nov 01 14:36:38.314835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:38.314752    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:38.316975 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:38.316944    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 14:36:38.317192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:38.317165    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 14:36:39.315845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:39.315804    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 14:36:39.316493 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:39.316461    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 14:36:40.320412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:40.319872    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 14:36:40.320412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:40.319935    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 14:36:41.317405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:41.314791    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 14:36:41.317405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:41.314866    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 14:36:42.319443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:42.317688    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 14:36:42.319443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:42.317747    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 14:36:43.313403 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:43.313346    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 01 14:36:43.313628 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:43.313445    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 01 14:36:44.318199 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:44.318164    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:44.318679 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:44.318482    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:44.319129 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:44.319113    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:36:45.506484 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.506456    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"]
Nov 01 14:36:45.510731 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.510706    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.617277 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.617246    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"]
Nov 01 14:36:45.627151 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627124    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.627289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627159    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.627289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627245    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.627289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627269    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.627480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627309    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.627480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627327    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.627586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627536    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904a66ff-67b2-4a86-8756-ceeb67668dff-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.627644 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627594    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-kube-api-access-fhptm\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.627644 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.627637    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729120    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729157    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729208    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729225    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729278    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904a66ff-67b2-4a86-8756-ceeb67668dff-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729321    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-kube-api-access-fhptm\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729358    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729730 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729509    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729730 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729539    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729768    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729863    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.729952 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.729922    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.730052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.730029    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.730139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.730117    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904a66ff-67b2-4a86-8756-ceeb67668dff-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.731747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.731725    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.732141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.732121    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.786703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.786641    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.786862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.786837    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-kube-api-access-fhptm\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:45.824582 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:45.824556    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:46.242493 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:36:46.242454    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904a66ff_67b2_4a86_8756_ceeb67668dff.slice/crio-a4b8c6f119a30120730d4a65edfbe2fbbadfcd31f998bd5a443360a455e70ede WatchSource:0}: Error finding container a4b8c6f119a30120730d4a65edfbe2fbbadfcd31f998bd5a443360a455e70ede: Status 404 returned error can't find the container with id a4b8c6f119a30120730d4a65edfbe2fbbadfcd31f998bd5a443360a455e70ede
Nov 01 14:36:46.244571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:46.244543    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"]
Nov 01 14:36:46.244945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:46.244914    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:36:46.245000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:46.244979    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:36:46.245040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:46.245006    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:36:46.384405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:46.383750    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" event={"ID":"904a66ff-67b2-4a86-8756-ceeb67668dff","Type":"ContainerStarted","Data":"0f8fafbace9dbcef93425f99fe02b139e8db5478e9d0930cd7a5c632f113f6ec"}
Nov 01 14:36:46.384405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:46.383812    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" event={"ID":"904a66ff-67b2-4a86-8756-ceeb67668dff","Type":"ContainerStarted","Data":"a4b8c6f119a30120730d4a65edfbe2fbbadfcd31f998bd5a443360a455e70ede"}
Nov 01 14:36:46.502769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:46.502659    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podStartSLOduration=1.502639595 podStartE2EDuration="1.502639595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:46.499808979 +0000 UTC m=+1146.665294277" watchObservedRunningTime="2026-11-01 14:36:46.502639595 +0000 UTC m=+1146.668124894"
Nov 01 14:36:46.825057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:46.825035    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:47.700411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:47.696329    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:36:47.826850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:47.826816    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:36:47.827228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:47.826870    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 14:36:48.838516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:48.838481    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:36:48.839111 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:48.839078    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 14:36:49.834807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:49.834768    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:36:49.835076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:49.835044    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 14:36:50.421542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:50.421494    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:36:50.480273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:50.480238    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:36:50.480555 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:50.480526    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:36:50.831869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:50.831833    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:36:50.832156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:50.832122    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 14:36:50.980750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:50.980700    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:36:51.067405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:51.067348    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:36:51.067698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:51.067655    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:36:51.825583 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:51.825545    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:36:51.826065 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:51.825597    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 14:36:52.825817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:52.825776    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:36:52.826258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:52.825846    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 01 14:36:53.863238 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:53.863127    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:53.863809 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:53.863757    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:36:53.864680 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:36:53.864658    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:37:05.148481 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:37:05.148446    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:37:05.455512 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.455441    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"]
Nov 01 14:37:05.459381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.459365    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.512803 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.512772    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"]
Nov 01 14:37:05.527827 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.527794    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-dshm\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.527962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.527834    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tls-certs\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.527962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.527906    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tmp-dir\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.527962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.527935    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-model-cache\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.528146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.527967    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-home\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.528146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.528089    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86784\" (UniqueName: \"kubernetes.io/projected/6a367b54-8394-4670-9a95-0ad8e9e08c0c-kube-api-access-86784\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.560224 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.560201    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"distinct-2-w2-007270-kserve-self-signed-certs\""
Nov 01 14:37:05.637410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.635182    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tmp-dir\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.637410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.635246    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-model-cache\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.637410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.635303    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-home\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.637410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.635441    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86784\" (UniqueName: \"kubernetes.io/projected/6a367b54-8394-4670-9a95-0ad8e9e08c0c-kube-api-access-86784\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.637410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.635589    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-dshm\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.637410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.635630    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tls-certs\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.660412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.638324    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-home\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.660412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.645717    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tmp-dir\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.660758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.646050    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-model-cache\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.660758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.647062    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tls-certs\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.660758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.649543    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-dshm\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.701977 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.701940    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86784\" (UniqueName: \"kubernetes.io/projected/6a367b54-8394-4670-9a95-0ad8e9e08c0c-kube-api-access-86784\") pod \"distinct-2-w2-007270-kserve-747bdf4bcd-924h8\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") " pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:05.772158 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:05.772085    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:06.329414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:06.323244    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"]
Nov 01 14:37:06.701260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:06.701220    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" event={"ID":"6a367b54-8394-4670-9a95-0ad8e9e08c0c","Type":"ContainerStarted","Data":"37ea0e83332f984730073fe1bf99fd224d6a3125d0635c44581683bfbf2d852d"}
Nov 01 14:37:06.701551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:06.701527    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" event={"ID":"6a367b54-8394-4670-9a95-0ad8e9e08c0c","Type":"ContainerStarted","Data":"ef433579c1e2162e9e69d05fdd8c1d1c5c79c538f2aff74f2ad09412fee6bfbe"}
Nov 01 14:37:06.927736 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:06.927702    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:06.987464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:06.986350    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podStartSLOduration=1.9863321329999999 podStartE2EDuration="1.986332133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:06.984204809 +0000 UTC m=+1167.149690107" watchObservedRunningTime="2026-11-01 14:37:06.986332133 +0000 UTC m=+1167.151817431"
Nov 01 14:37:07.929533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:07.929473    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:08.001519 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:08.001477    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 14:37:08.184604 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:08.184507    5423 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Nov 01 14:37:08.390410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:08.383494    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 01 14:37:15.772453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:15.772410    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:15.784878 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:15.784858    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:15.869062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:15.869024    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:37:19.962303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:19.962230    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:37:22.689557 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:22.689499    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:22.861531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:22.861483    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:22.955462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:22.952489    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:23.008411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:22.992526    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:23.509929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:23.509886    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:35.149609 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:37:35.149570    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:37:40.293287 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:40.293260    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:37:40.300094 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:40.300072    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:37:41.713551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:41.713503    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:42.719323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:42.719282    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 14:37:42.720362 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:42.720341    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:37:42.744674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:42.744581    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:37:46.021646 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.021610    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"]
Nov 01 14:37:46.027648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.027623    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.041673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.041643    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"]
Nov 01 14:37:46.045686 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.045661    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.082240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.082205    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"]
Nov 01 14:37:46.126534 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.126500    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"]
Nov 01 14:37:46.127697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.127668    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.127978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.127954    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.128223 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128190    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.128313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128237    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5417f6-4949-4efd-8429-69374e520c77-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.128365 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128312    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.128365 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128338    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82556296-1a82-4ac7-828c-b8a223aa3fd7-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.128494 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128417    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.128549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128505    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.128549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128534    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tz7r\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-kube-api-access-7tz7r\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.128654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128565    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.128654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128583    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.128654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128608    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.128654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128625    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.128654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128651    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.128916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128705    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.128916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128742    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5417f6-4949-4efd-8429-69374e520c77-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.128916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128767    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.128916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.128812    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j978v\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-kube-api-access-j978v\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.230345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230312    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.230583 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230376    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.230583 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230423    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5417f6-4949-4efd-8429-69374e520c77-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.230583 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230531    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.230754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230589    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j978v\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-kube-api-access-j978v\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.230754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230625    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.230754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230650    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.230754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230675    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.230754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230711    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5417f6-4949-4efd-8429-69374e520c77-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.230754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230751    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230772    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82556296-1a82-4ac7-828c-b8a223aa3fd7-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230806    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230818    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.231083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230944    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.231083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230952    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.230991    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tz7r\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-kube-api-access-7tz7r\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.231083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231040    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.231083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231073    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231620 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231105    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.231620 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231119    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231620 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231154    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231620 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231380    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231620 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231508    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82556296-1a82-4ac7-828c-b8a223aa3fd7-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.231871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231827    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5417f6-4949-4efd-8429-69374e520c77-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.232009 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.231987    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.232070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.232049    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.232108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.232075    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.232189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.232173    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.233615 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.233592    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.233708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.233657    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5417f6-4949-4efd-8429-69374e520c77-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.233767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.233734    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.233922 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.233904    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.287599 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.287539    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.287976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.287955    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.288088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.288065    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j978v\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-kube-api-access-j978v\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.288212 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.288196    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tz7r\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-kube-api-access-7tz7r\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.342295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.342258    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:46.363719 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.363650    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:46.769817 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:37:46.769785    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5417f6_4949_4efd_8429_69374e520c77.slice/crio-184813d5e5dc14005148a01b8ef434a29b8f901e87daf656ee97dae366c6baae WatchSource:0}: Error finding container 184813d5e5dc14005148a01b8ef434a29b8f901e87daf656ee97dae366c6baae: Status 404 returned error can't find the container with id 184813d5e5dc14005148a01b8ef434a29b8f901e87daf656ee97dae366c6baae
Nov 01 14:37:46.770720 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:37:46.770695    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82556296_1a82_4ac7_828c_b8a223aa3fd7.slice/crio-1a9a962f522ed3d934eab79b86645a7a3d6dbc04b6b20b93b555ddff9930c231 WatchSource:0}: Error finding container 1a9a962f522ed3d934eab79b86645a7a3d6dbc04b6b20b93b555ddff9930c231: Status 404 returned error can't find the container with id 1a9a962f522ed3d934eab79b86645a7a3d6dbc04b6b20b93b555ddff9930c231
Nov 01 14:37:46.770854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.770762    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"]
Nov 01 14:37:46.772339 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.772310    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:46.772488 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.772367    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:46.772488 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.772438    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:46.777323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.777295    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"]
Nov 01 14:37:46.781665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.781627    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:46.781905 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.781878    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:46.783441 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:46.783410    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:47.266736 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.266702    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" event={"ID":"82556296-1a82-4ac7-828c-b8a223aa3fd7","Type":"ContainerStarted","Data":"c9259cdf042adca33c63acd0c8e6937b3b8dcf1494920d9afd61f66e65ec3350"}
Nov 01 14:37:47.267207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.266758    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" event={"ID":"82556296-1a82-4ac7-828c-b8a223aa3fd7","Type":"ContainerStarted","Data":"1a9a962f522ed3d934eab79b86645a7a3d6dbc04b6b20b93b555ddff9930c231"}
Nov 01 14:37:47.268486 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.268457    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" event={"ID":"2c5417f6-4949-4efd-8429-69374e520c77","Type":"ContainerStarted","Data":"2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9"}
Nov 01 14:37:47.268606 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.268506    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" event={"ID":"2c5417f6-4949-4efd-8429-69374e520c77","Type":"ContainerStarted","Data":"184813d5e5dc14005148a01b8ef434a29b8f901e87daf656ee97dae366c6baae"}
Nov 01 14:37:47.343362 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.343325    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:47.345017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.344989    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:47.345188 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.345034    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:47.363775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.363747    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:47.365273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.365250    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:47.365400 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.365294    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:47.384109 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.384050    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podStartSLOduration=2.384033409 podStartE2EDuration="2.384033409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:47.383110596 +0000 UTC m=+1207.548595895" watchObservedRunningTime="2026-11-01 14:37:47.384033409 +0000 UTC m=+1207.549518712"
Nov 01 14:37:47.500898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:47.500837    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podStartSLOduration=2.500820083 podStartE2EDuration="2.500820083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:47.498999034 +0000 UTC m=+1207.664484331" watchObservedRunningTime="2026-11-01 14:37:47.500820083 +0000 UTC m=+1207.666305381"
Nov 01 14:37:48.344410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:48.344045    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:48.344410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:48.344121    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:48.364476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:48.364436    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:48.364611 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:48.364502    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:49.346759 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:49.343744    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:49.346759 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:49.343810    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:49.366422 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:49.365504    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:49.366422 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:49.365571    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:50.343412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:50.342782    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:50.343412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:50.342848    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:50.364103 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:50.364069    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:50.364480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:50.364115    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:50.858083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:50.857235    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:37:51.350593 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:51.350554    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:51.350778 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:51.350611    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:51.367434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:51.366260    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:51.367434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:51.366330    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:52.350852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:52.350805    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:52.351168 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:52.351136    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:52.370708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:52.370676    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:52.371297 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:52.371264    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:52.763736 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:52.763631    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 01 14:37:52.920599 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:52.920527    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.033463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.006518    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.351411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.342765    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:53.351411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.342827    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:53.367795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.367757    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:53.368067 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.368040    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:53.415853 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.415480    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:53.815463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.803487    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:53.815463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.814872    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:37:53.899410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.896633    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.988998 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.928501    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.988998 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:53.961430    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:37:54.014446 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.013642    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:54.297842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.296102    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:37:54.422926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.418759    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:54.422926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.419413    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:37:54.450974 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.433978    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:54.450974 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.434040    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:54.464405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.462536    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:54.464405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.462619    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:54.779162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.779133    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 14:37:54.859464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.844224    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:54.976046 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:54.976014    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 14:37:55.053467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.045763    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:37:55.079028 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.078973    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:37:55.343177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.343135    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 01 14:37:55.343463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.343202    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 01 14:37:55.364767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.364718    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:37:55.365043 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.365004    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 14:37:55.899336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.899290    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"]
Nov 01 14:37:55.904943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.904918    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:55.959836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:55.959802    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c9b0827c-openshift-default-dockercfg-4c75l\""
Nov 01 14:37:56.012410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.011137    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"]
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049037    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-credential-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049097    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz69p\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-kube-api-access-xz69p\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049145    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40036455-b3f2-42fa-9aec-c29413f37802-istiod-ca-cert\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049296    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049373    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-certs\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049450    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-envoy\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049490    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-data\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049514    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40036455-b3f2-42fa-9aec-c29413f37802-istio-podinfo\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.051437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.049569    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-istio-token\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.150941    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-istio-token\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151322 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.151293    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-credential-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.151350    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz69p\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-kube-api-access-xz69p\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.151421    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40036455-b3f2-42fa-9aec-c29413f37802-istiod-ca-cert\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151610 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.151576    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151676 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.151656    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-certs\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.151761    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-envoy\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.151806    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-data\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.151858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.151838    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40036455-b3f2-42fa-9aec-c29413f37802-istio-podinfo\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.152078 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.152055    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.152810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.152786    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-certs\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.155320 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.153013    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40036455-b3f2-42fa-9aec-c29413f37802-istiod-ca-cert\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.155320 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.153333    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-data\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.155320 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.154490    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-credential-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.155835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.155812    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-envoy\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.156188 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.156164    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40036455-b3f2-42fa-9aec-c29413f37802-istio-podinfo\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.213747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.213673    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz69p\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-kube-api-access-xz69p\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.215511 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.215481    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-istio-token\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.226971 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.226541    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:56.660503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.660349    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:56.661230 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.661208    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:56.662280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.662255    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:37:56.904355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.904292    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:56.904830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.904373    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:56.904830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.904432    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:37:56.909536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.908229    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"]
Nov 01 14:37:56.917202 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.917173    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:56.918501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.918479    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:56.920446 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:56.920420    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:37:57.444702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:57.444656    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" event={"ID":"40036455-b3f2-42fa-9aec-c29413f37802","Type":"ContainerStarted","Data":"4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0"}
Nov 01 14:37:57.444903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:57.444719    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" event={"ID":"40036455-b3f2-42fa-9aec-c29413f37802","Type":"ContainerStarted","Data":"d817a27653e10460061ed5ae81c638ae343d1452fe679c5b7f7cd0c98913d44c"}
Nov 01 14:37:57.561810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:57.561762    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podStartSLOduration=2.561747601 podStartE2EDuration="2.561747601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:57.56136962 +0000 UTC m=+1217.726854918" watchObservedRunningTime="2026-11-01 14:37:57.561747601 +0000 UTC m=+1217.727232899"
Nov 01 14:37:58.227940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:58.227909    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:58.233302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:58.233279    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:58.451904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:58.451865    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:37:58.452438 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:37:58.452419    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:38:03.913409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:03.912302    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"]
Nov 01 14:38:03.940837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:03.940537    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:03.980345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:03.980314    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"]
Nov 01 14:38:04.002324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.002288    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c9b0827c\""
Nov 01 14:38:04.003009 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.002729    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c9b0827c-dockercfg-smdpw\""
Nov 01 14:38:04.017941 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.017751    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnlcx\" (UniqueName: \"kubernetes.io/projected/1c02c67d-5add-4ff3-801a-295885e693b9-kube-api-access-pnlcx\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd\" (UID: \"1c02c67d-5add-4ff3-801a-295885e693b9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:04.017941 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.017826    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c02c67d-5add-4ff3-801a-295885e693b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd\" (UID: \"1c02c67d-5add-4ff3-801a-295885e693b9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:04.198805 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.148286    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnlcx\" (UniqueName: \"kubernetes.io/projected/1c02c67d-5add-4ff3-801a-295885e693b9-kube-api-access-pnlcx\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd\" (UID: \"1c02c67d-5add-4ff3-801a-295885e693b9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:04.218444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.218410    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c02c67d-5add-4ff3-801a-295885e693b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd\" (UID: \"1c02c67d-5add-4ff3-801a-295885e693b9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:04.219964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.219936    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c02c67d-5add-4ff3-801a-295885e693b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd\" (UID: \"1c02c67d-5add-4ff3-801a-295885e693b9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:04.246937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.245960    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnlcx\" (UniqueName: \"kubernetes.io/projected/1c02c67d-5add-4ff3-801a-295885e693b9-kube-api-access-pnlcx\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd\" (UID: \"1c02c67d-5add-4ff3-801a-295885e693b9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:04.353934 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.353904    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:04.723219 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:38:04.723181    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c02c67d_5add_4ff3_801a_295885e693b9.slice/crio-ad43c1dbab03af746a94117574a9f6dc172827e5c12eb2d1b9428cf11d84a083 WatchSource:0}: Error finding container ad43c1dbab03af746a94117574a9f6dc172827e5c12eb2d1b9428cf11d84a083: Status 404 returned error can't find the container with id ad43c1dbab03af746a94117574a9f6dc172827e5c12eb2d1b9428cf11d84a083
Nov 01 14:38:04.725656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:04.725633    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"]
Nov 01 14:38:05.149419 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:38:05.149358    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:38:05.492999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:05.492913    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd" event={"ID":"1c02c67d-5add-4ff3-801a-295885e693b9","Type":"ContainerStarted","Data":"14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623"}
Nov 01 14:38:05.492999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:05.492989    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd" event={"ID":"1c02c67d-5add-4ff3-801a-295885e693b9","Type":"ContainerStarted","Data":"ad43c1dbab03af746a94117574a9f6dc172827e5c12eb2d1b9428cf11d84a083"}
Nov 01 14:38:05.493231 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:05.493022    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:05.607887 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:05.607833    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd" podStartSLOduration=2.6078185879999998 podStartE2EDuration="2.607818588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:05.605258225 +0000 UTC m=+1225.770743523" watchObservedRunningTime="2026-11-01 14:38:05.607818588 +0000 UTC m=+1225.773303919"
Nov 01 14:38:11.319414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:11.318800    5423 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Nov 01 14:38:11.798268 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:11.797486    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 01 14:38:16.502243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:16.502213    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:38:24.550281 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.550216    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"]
Nov 01 14:38:24.555295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.555272    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.608528 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.608497    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-84ded4-openshift-default-dockercfg-td8dn\""
Nov 01 14:38:24.662954 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.662918    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"]
Nov 01 14:38:24.726672 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.726623    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.726859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.726740    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.726859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.726783    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.726947 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.726927    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.726983 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.726952    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.727020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.726987    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.727056 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.727037    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c697p\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-kube-api-access-c697p\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.727120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.727098    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.727164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.727153    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837429    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837485    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837580    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837612    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c697p\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-kube-api-access-c697p\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837665    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837729    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837853    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837909    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.837949    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.838513    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.839121    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.839784    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.845409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.840297    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.856410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.854919    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.896404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.889904    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.896404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.890183    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.917473 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.912033    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:24.917473 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:24.913584    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c697p\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-kube-api-access-c697p\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:25.170969 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:25.170883    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:25.591810 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:38:25.591777    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcb770b_5f04_44f0_9cae_d882ddd526a8.slice/crio-eb5220112ae19eca93440809281074efddf592262fae2b522e5871c63661dde0 WatchSource:0}: Error finding container eb5220112ae19eca93440809281074efddf592262fae2b522e5871c63661dde0: Status 404 returned error can't find the container with id eb5220112ae19eca93440809281074efddf592262fae2b522e5871c63661dde0
Nov 01 14:38:25.593506 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:25.593486    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"]
Nov 01 14:38:25.594232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:25.594201    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:38:25.594299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:25.594260    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:38:25.594299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:25.594288    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:38:25.660186 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:25.660158    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" event={"ID":"4fcb770b-5f04-44f0-9cae-d882ddd526a8","Type":"ContainerStarted","Data":"eb5220112ae19eca93440809281074efddf592262fae2b522e5871c63661dde0"}
Nov 01 14:38:26.668235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:26.668200    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" event={"ID":"4fcb770b-5f04-44f0-9cae-d882ddd526a8","Type":"ContainerStarted","Data":"b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c"}
Nov 01 14:38:26.835605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:26.835545    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" podStartSLOduration=2.835531306 podStartE2EDuration="2.835531306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:26.834031861 +0000 UTC m=+1246.999517158" watchObservedRunningTime="2026-11-01 14:38:26.835531306 +0000 UTC m=+1247.001016604"
Nov 01 14:38:27.171944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:27.171911    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:27.177724 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:27.177698    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:27.674857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:27.674827    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:27.675841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:27.675821    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:38:30.800155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:30.800116    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:38:31.366412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.362938    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"]
Nov 01 14:38:31.388408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.387510    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.438990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.438953    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"]
Nov 01 14:38:31.501902 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.501864    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.502188 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.502142    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.502351 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.502329    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.502556 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.502537    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.504547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.504522    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.506053 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.505907    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.506053 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.505959    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb9s2\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-kube-api-access-jb9s2\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.506270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.506060    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.506270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.506108    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.606963 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.606923    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.607252 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.607217    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.607653 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.607629    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.607835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.607816    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.608002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.607985    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.608201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.608181    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.608354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.608335    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.608553 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.608527    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.608703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.608684    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb9s2\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-kube-api-access-jb9s2\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.610059 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.610034    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.611866 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.611561    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.611866 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.611825    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.613258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.613230    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.613487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.613430    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.615861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.615817    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.665348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.665250    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.669493 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.669462    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.669623 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.669561    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb9s2\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-kube-api-access-jb9s2\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:31.812409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:31.808906    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:32.774420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:32.773455    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"]
Nov 01 14:38:33.001297 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:33.001163    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:38:33.002029 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:33.002002    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:38:33.002315 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:33.002194    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:38:33.063456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:33.059511    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:33.865849 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:33.865808    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" event={"ID":"1ba6a000-311b-4f1f-97f6-298bb0e2f87e","Type":"ContainerStarted","Data":"658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6"}
Nov 01 14:38:33.887070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:33.887038    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" event={"ID":"1ba6a000-311b-4f1f-97f6-298bb0e2f87e","Type":"ContainerStarted","Data":"d990b0652ec752b5f2583e0f255a28575ab49477477f739699f0b7c5053bfc18"}
Nov 01 14:38:34.279516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.271878    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podStartSLOduration=3.27185447 podStartE2EDuration="3.27185447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:34.171536441 +0000 UTC m=+1254.337021740" watchObservedRunningTime="2026-11-01 14:38:34.27185447 +0000 UTC m=+1254.437339769"
Nov 01 14:38:34.573352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.572640    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"]
Nov 01 14:38:34.592080 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.592049    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:34.717225 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.717189    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"]
Nov 01 14:38:34.721865 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.721834    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-84ded4\""
Nov 01 14:38:34.742465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.732335    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8385d2-567e-4897-a1e7-758dc183e95b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp\" (UID: \"6f8385d2-567e-4897-a1e7-758dc183e95b\") " pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:34.764472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.733155    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-84ded4-dockercfg-5t7gh\""
Nov 01 14:38:34.800543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.800492    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:34.818419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.809444    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpzj\" (UniqueName: \"kubernetes.io/projected/6f8385d2-567e-4897-a1e7-758dc183e95b-kube-api-access-zwpzj\") pod \"payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp\" (UID: \"6f8385d2-567e-4897-a1e7-758dc183e95b\") " pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:34.818419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.811679    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:34.818419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.811791    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 14:38:34.818419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.811831    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 14:38:34.924407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.924295    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpzj\" (UniqueName: \"kubernetes.io/projected/6f8385d2-567e-4897-a1e7-758dc183e95b-kube-api-access-zwpzj\") pod \"payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp\" (UID: \"6f8385d2-567e-4897-a1e7-758dc183e95b\") " pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:34.924808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.924783    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8385d2-567e-4897-a1e7-758dc183e95b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp\" (UID: \"6f8385d2-567e-4897-a1e7-758dc183e95b\") " pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:34.940407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.925903    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8385d2-567e-4897-a1e7-758dc183e95b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp\" (UID: \"6f8385d2-567e-4897-a1e7-758dc183e95b\") " pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:35.152407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:34.941284    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 14:38:35.266684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:35.266601    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpzj\" (UniqueName: \"kubernetes.io/projected/6f8385d2-567e-4897-a1e7-758dc183e95b-kube-api-access-zwpzj\") pod \"payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp\" (UID: \"6f8385d2-567e-4897-a1e7-758dc183e95b\") " pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:35.361361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:35.361098    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:35.468804 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:38:35.468732    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:38:35.816411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:35.812867    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 14:38:36.262465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:36.239570    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 14:38:36.951478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:36.950442    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 14:38:36.951478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:36.950506    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 14:38:37.276607 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:37.276513    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp" event={"ID":"6f8385d2-567e-4897-a1e7-758dc183e95b","Type":"ContainerStarted","Data":"0176d967effba21e60c50f78a2b607b5ecf5c62898bae01f9b2562d5cfc8a0c9"}
Nov 01 14:38:37.331645 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:37.331605    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"]
Nov 01 14:38:37.819412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:37.815764    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 14:38:37.819412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:37.815840    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 14:38:38.283796 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:38.283701    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp" event={"ID":"6f8385d2-567e-4897-a1e7-758dc183e95b","Type":"ContainerStarted","Data":"c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4"}
Nov 01 14:38:38.284196 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:38.283803    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:38:38.418409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:38.417772    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp" podStartSLOduration=4.417753638 podStartE2EDuration="4.417753638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:38.416931613 +0000 UTC m=+1258.582416913" watchObservedRunningTime="2026-11-01 14:38:38.417753638 +0000 UTC m=+1258.583238937"
Nov 01 14:38:38.812412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:38.811190    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 14:38:38.812412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:38.811249    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 14:38:39.813411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:39.809735    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 14:38:39.813411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:39.809798    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 14:38:40.814993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:40.814957    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 14:38:40.816058 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:40.815641    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 14:38:41.810447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:41.810366    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 01 14:38:41.810997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:41.810474    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 01 14:38:42.868945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:42.868812    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:42.869824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:42.869794    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:42.871593 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:42.871570    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:38:49.293246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:38:49.293212    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:39:00.832536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.832499    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"]
Nov 01 14:39:00.837186 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.837164    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.887585 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.887554    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"]
Nov 01 14:39:00.892163 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.892129    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-routing-overlay-volume\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.892306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.892177    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-provider-credentials\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.892451 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.892423    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-plugins-config-volume\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.892588 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.892568    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9867\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-kube-api-access-x9867\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.935008 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.934979    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 14:39:00.935176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.935047    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 14:39:00.935176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.935083    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 14:39:00.935176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.935121    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee13fb54a62a\""
Nov 01 14:39:00.939544 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.939516    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 14:39:00.939662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.939597    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vn9m6\""
Nov 01 14:39:00.994230 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.994188    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9867\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-kube-api-access-x9867\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.994439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.994323    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-routing-overlay-volume\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.994439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.994353    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-provider-credentials\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.994568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.994532    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-plugins-config-volume\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.995162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.995134    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-routing-overlay-volume\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.995300 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.995282    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-plugins-config-volume\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:00.997301 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:00.997276    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-provider-credentials\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:01.053816 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:01.053786    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9867\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-kube-api-access-x9867\") pod \"payload-processing-external-model-5dfd5db6bc-nhjdh\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") " pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:01.153962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:01.153892    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:01.536615 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:39:01.536564    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcec320_e05c_4739_abf1_a9acce274458.slice/crio-851a2472d30faaaa2d7bb336d02fd8a61684ce50e4acfe7608c6c825cfc12817 WatchSource:0}: Error finding container 851a2472d30faaaa2d7bb336d02fd8a61684ce50e4acfe7608c6c825cfc12817: Status 404 returned error can't find the container with id 851a2472d30faaaa2d7bb336d02fd8a61684ce50e4acfe7608c6c825cfc12817
Nov 01 14:39:01.538865 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:01.538834    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"]
Nov 01 14:39:01.566187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:01.566161    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh" event={"ID":"fdcec320-e05c-4739-abf1-a9acce274458","Type":"ContainerStarted","Data":"851a2472d30faaaa2d7bb336d02fd8a61684ce50e4acfe7608c6c825cfc12817"}
Nov 01 14:39:02.574403 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:02.574339    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh" event={"ID":"fdcec320-e05c-4739-abf1-a9acce274458","Type":"ContainerStarted","Data":"293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d"}
Nov 01 14:39:02.575345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:02.574510    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:02.688348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:02.688283    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh" podStartSLOduration=2.6882680839999997 podStartE2EDuration="2.688268084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:02.687248709 +0000 UTC m=+1282.852734008" watchObservedRunningTime="2026-11-01 14:39:02.688268084 +0000 UTC m=+1282.853753384"
Nov 01 14:39:04.587855 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:04.587823    5423 generic.go:356] "Generic (PLEG): container finished" podID="40036455-b3f2-42fa-9aec-c29413f37802" containerID="4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0" exitCode=0
Nov 01 14:39:04.588265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:04.587894    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" event={"ID":"40036455-b3f2-42fa-9aec-c29413f37802","Type":"ContainerDied","Data":"4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0"}
Nov 01 14:39:04.588318 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:04.588302    5423 scope.go:122] "RemoveContainer" containerID="4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0"
Nov 01 14:39:04.589720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:04.589694    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:04.589779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:04.589761    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:04.589855 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:04.589793    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:05.150182 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:39:05.150146    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:39:05.242207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.242177    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"]
Nov 01 14:39:05.245225 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.245209    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.297707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.297682    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-84ded4-serving-cert\""
Nov 01 14:39:05.333875 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.333848    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"]
Nov 01 14:39:05.351524 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.351493    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6b8eb1-df62-4619-b07b-97a16b56afba-openshift-service-ca\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.351664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.351575    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.351664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.351638    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-metrics-certs\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.351796 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.351727    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhd4\" (UniqueName: \"kubernetes.io/projected/4a6b8eb1-df62-4619-b07b-97a16b56afba-kube-api-access-5jhd4\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.452893 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.452811    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhd4\" (UniqueName: \"kubernetes.io/projected/4a6b8eb1-df62-4619-b07b-97a16b56afba-kube-api-access-5jhd4\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.453054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.452920    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6b8eb1-df62-4619-b07b-97a16b56afba-openshift-service-ca\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.453054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.452955    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.453054 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.453008    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-metrics-certs\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.453244 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:39:05.453139    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-serving-cert: secret "maas-api-e2e-api-a-84ded4-serving-cert" not found
Nov 01 14:39:05.453244 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:39:05.453231    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls podName:4a6b8eb1-df62-4619-b07b-97a16b56afba nodeName:}" failed. No retries permitted until 2026-11-01 14:39:05.953207667 +0000 UTC m=+1286.118692945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls") pod "maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n" (UID: "4a6b8eb1-df62-4619-b07b-97a16b56afba") : secret "maas-api-e2e-api-a-84ded4-serving-cert" not found
Nov 01 14:39:05.453694 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.453674    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6b8eb1-df62-4619-b07b-97a16b56afba-openshift-service-ca\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.455471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.455451    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-metrics-certs\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.545411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.533183    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhd4\" (UniqueName: \"kubernetes.io/projected/4a6b8eb1-df62-4619-b07b-97a16b56afba-kube-api-access-5jhd4\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:05.620410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:05.609913    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" event={"ID":"40036455-b3f2-42fa-9aec-c29413f37802","Type":"ContainerStarted","Data":"9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789"}
Nov 01 14:39:06.042417 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.041427    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:06.177407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.170077    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls\") pod \"maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:06.177407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.176355    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:06.246409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.227911    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:39:06.246409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.228056    5423 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:39:06.246409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.228138    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 14:39:06.391424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.389948    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"]
Nov 01 14:39:06.442121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.415208    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"]
Nov 01 14:39:06.442121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.415317    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:06.479217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.477652    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-466a6e\""
Nov 01 14:39:06.479217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.477795    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-466a6e-dockercfg-8clxc\""
Nov 01 14:39:06.662945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.620401    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5927d613-42a6-4a03-a2dd-ad33d10bbd54-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb\" (UID: \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\") " pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:06.662945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.620522    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79cx\" (UniqueName: \"kubernetes.io/projected/5927d613-42a6-4a03-a2dd-ad33d10bbd54-kube-api-access-h79cx\") pod \"payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb\" (UID: \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\") " pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:06.727903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.689106    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"]
Nov 01 14:39:06.727903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.695528    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:06.727903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.723341    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5927d613-42a6-4a03-a2dd-ad33d10bbd54-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb\" (UID: \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\") " pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:06.727903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.724049    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79cx\" (UniqueName: \"kubernetes.io/projected/5927d613-42a6-4a03-a2dd-ad33d10bbd54-kube-api-access-h79cx\") pod \"payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb\" (UID: \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\") " pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:06.727903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.725528    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5927d613-42a6-4a03-a2dd-ad33d10bbd54-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb\" (UID: \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\") " pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:06.776078 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.776038    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"]
Nov 01 14:39:06.785217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.785170    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79cx\" (UniqueName: \"kubernetes.io/projected/5927d613-42a6-4a03-a2dd-ad33d10bbd54-kube-api-access-h79cx\") pod \"payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb\" (UID: \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\") " pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:06.788516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.787921    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:06.830597 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.829121    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2023ad-75c9-4d6b-9b14-9186166a857a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q\" (UID: \"9b2023ad-75c9-4d6b-9b14-9186166a857a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:06.830597 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.829181    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p27xw\" (UniqueName: \"kubernetes.io/projected/9b2023ad-75c9-4d6b-9b14-9186166a857a-kube-api-access-p27xw\") pod \"payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q\" (UID: \"9b2023ad-75c9-4d6b-9b14-9186166a857a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:06.930364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.930280    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2023ad-75c9-4d6b-9b14-9186166a857a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q\" (UID: \"9b2023ad-75c9-4d6b-9b14-9186166a857a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:06.930364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.930331    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27xw\" (UniqueName: \"kubernetes.io/projected/9b2023ad-75c9-4d6b-9b14-9186166a857a-kube-api-access-p27xw\") pod \"payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q\" (UID: \"9b2023ad-75c9-4d6b-9b14-9186166a857a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:06.931125 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.931097    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2023ad-75c9-4d6b-9b14-9186166a857a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q\" (UID: \"9b2023ad-75c9-4d6b-9b14-9186166a857a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:06.966228 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:39:06.966197    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6b8eb1_df62_4619_b07b_97a16b56afba.slice/crio-bcc31bba1a4aee315c997f40c792d8550de2453b0723bde8e6f27e3adad38e28 WatchSource:0}: Error finding container bcc31bba1a4aee315c997f40c792d8550de2453b0723bde8e6f27e3adad38e28: Status 404 returned error can't find the container with id bcc31bba1a4aee315c997f40c792d8550de2453b0723bde8e6f27e3adad38e28
Nov 01 14:39:06.967718 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.967698    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:39:06.968852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.968833    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"]
Nov 01 14:39:06.987363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:06.987339    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27xw\" (UniqueName: \"kubernetes.io/projected/9b2023ad-75c9-4d6b-9b14-9186166a857a-kube-api-access-p27xw\") pod \"payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q\" (UID: \"9b2023ad-75c9-4d6b-9b14-9186166a857a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:07.022764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.022733    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:07.193233 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:39:07.193205    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5927d613_42a6_4a03_a2dd_ad33d10bbd54.slice/crio-dfb9224ed8291ec7f2eee815f4726cd010230a3eeea72b20733b218a136398ef WatchSource:0}: Error finding container dfb9224ed8291ec7f2eee815f4726cd010230a3eeea72b20733b218a136398ef: Status 404 returned error can't find the container with id dfb9224ed8291ec7f2eee815f4726cd010230a3eeea72b20733b218a136398ef
Nov 01 14:39:07.220062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.220015    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"]
Nov 01 14:39:07.228955 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.228932    5423 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:39:07.229088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.228983    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 14:39:07.391103 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:39:07.391062    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2023ad_75c9_4d6b_9b14_9186166a857a.slice/crio-67dcba62aedddb37ccb5f51d2fcbbc8c33663e4fb6f99a3e503eb62a06a6285c WatchSource:0}: Error finding container 67dcba62aedddb37ccb5f51d2fcbbc8c33663e4fb6f99a3e503eb62a06a6285c: Status 404 returned error can't find the container with id 67dcba62aedddb37ccb5f51d2fcbbc8c33663e4fb6f99a3e503eb62a06a6285c
Nov 01 14:39:07.393447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.393422    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"]
Nov 01 14:39:07.725318 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.725211    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q" event={"ID":"9b2023ad-75c9-4d6b-9b14-9186166a857a","Type":"ContainerStarted","Data":"b161c4cd51c87bad8149b1d7cb36e237732a83463df7335c48c332ff381461b1"}
Nov 01 14:39:07.725318 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.725286    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q" event={"ID":"9b2023ad-75c9-4d6b-9b14-9186166a857a","Type":"ContainerStarted","Data":"67dcba62aedddb37ccb5f51d2fcbbc8c33663e4fb6f99a3e503eb62a06a6285c"}
Nov 01 14:39:07.725318 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.725317    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:07.726804 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.726779    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n" event={"ID":"4a6b8eb1-df62-4619-b07b-97a16b56afba","Type":"ContainerStarted","Data":"bcc31bba1a4aee315c997f40c792d8550de2453b0723bde8e6f27e3adad38e28"}
Nov 01 14:39:07.728443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.728418    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb" event={"ID":"5927d613-42a6-4a03-a2dd-ad33d10bbd54","Type":"ContainerStarted","Data":"6f9e148ea1bcadf6fe6f0b27963db9da54d822fa5ed0895e62d1cafdaf5d76b3"}
Nov 01 14:39:07.728523 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.728465    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb" event={"ID":"5927d613-42a6-4a03-a2dd-ad33d10bbd54","Type":"ContainerStarted","Data":"dfb9224ed8291ec7f2eee815f4726cd010230a3eeea72b20733b218a136398ef"}
Nov 01 14:39:07.728523 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.728491    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:07.840758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.840712    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q" podStartSLOduration=1.840697394 podStartE2EDuration="1.840697394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:07.838430497 +0000 UTC m=+1288.003915796" watchObservedRunningTime="2026-11-01 14:39:07.840697394 +0000 UTC m=+1288.006182692"
Nov 01 14:39:07.971414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:07.970310    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb" podStartSLOduration=1.970289484 podStartE2EDuration="1.970289484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:07.95123019 +0000 UTC m=+1288.116715489" watchObservedRunningTime="2026-11-01 14:39:07.970289484 +0000 UTC m=+1288.135774783"
Nov 01 14:39:08.236680 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:08.236644    5423 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:39:08.236972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:08.236941    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 14:39:08.423891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:08.423851    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:39:08.739181 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:08.738611    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n" event={"ID":"4a6b8eb1-df62-4619-b07b-97a16b56afba","Type":"ContainerStarted","Data":"150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207"}
Nov 01 14:39:08.739181 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:08.739075    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:08.856726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:08.856672    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n" podStartSLOduration=2.99044276 podStartE2EDuration="3.856658161s" totalImagesPullingTime="866.215401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:08.853479609 +0000 UTC m=+1289.018964908" watchObservedRunningTime="2026-11-01 14:39:08.856658161 +0000 UTC m=+1289.022143459"
Nov 01 14:39:09.228288 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:09.227702    5423 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:39:09.228288 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:09.227762    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 14:39:10.234409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:10.233809    5423 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:39:10.234409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:10.233870    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 14:39:10.904411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:10.903977    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:39:11.052744 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.052697    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:39:11.091411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.078854    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:39:11.247451 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.241879    5423 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:39:11.247451 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.241937    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 14:39:11.302056 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.302007    5423 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:39:11.782649 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.782600    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:39:11.907527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.907464    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:39:11.908579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.908554    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:39:11.981417 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:11.977022    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:39:12.088410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.072155    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 01 14:39:12.139630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.106854    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:39:12.169411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.134570    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:39:12.169411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.162494    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:39:12.499461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.493031    5423 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:39:12.499461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.493105    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 01 14:39:12.635227 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.588117    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 14:39:12.635227 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.588187    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 14:39:12.635227 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:12.589201    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 14:39:13.336496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:13.336459    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:39:13.338197 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:13.338173    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:39:13.340146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:13.340121    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:39:13.582797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:13.582760    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh" podUID="fdcec320-e05c-4739-abf1-a9acce274458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:39:14.794114 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:14.794081    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"]
Nov 01 14:39:14.795237 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:14.795208    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh" podUID="fdcec320-e05c-4739-abf1-a9acce274458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:39:16.109257 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:16.109071    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:39:18.741133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:18.741097    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:39:18.741742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:18.741720    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:39:19.795324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:19.795283    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh" podUID="fdcec320-e05c-4739-abf1-a9acce274458" containerName="payload-processing" containerID="cri-o://293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d" gracePeriod=25
Nov 01 14:39:20.057131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.057107    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:20.125912 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.125885    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-kube-api-access-x9867\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-kube-api-access-x9867\") pod \"fdcec320-e05c-4739-abf1-a9acce274458\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") "
Nov 01 14:39:20.126056 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.125945    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-provider-credentials\") pod \"fdcec320-e05c-4739-abf1-a9acce274458\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") "
Nov 01 14:39:20.126098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.126085    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-routing-overlay-volume\") pod \"fdcec320-e05c-4739-abf1-a9acce274458\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") "
Nov 01 14:39:20.126213 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.126195    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-plugins-config-volume\") pod \"fdcec320-e05c-4739-abf1-a9acce274458\" (UID: \"fdcec320-e05c-4739-abf1-a9acce274458\") "
Nov 01 14:39:20.126450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.126424    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-routing-overlay-volume" pod "fdcec320-e05c-4739-abf1-a9acce274458" (UID: "fdcec320-e05c-4739-abf1-a9acce274458"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:39:20.126525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.126504    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-plugins-config-volume" pod "fdcec320-e05c-4739-abf1-a9acce274458" (UID: "fdcec320-e05c-4739-abf1-a9acce274458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:39:20.126727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.126713    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:39:20.126767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.126730    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcec320-e05c-4739-abf1-a9acce274458-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:39:20.128940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.128919    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-kube-api-access-x9867" pod "fdcec320-e05c-4739-abf1-a9acce274458" (UID: "fdcec320-e05c-4739-abf1-a9acce274458"). InnerVolumeSpecName "kube-api-access-x9867". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:39:20.128940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.128927    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-provider-credentials" pod "fdcec320-e05c-4739-abf1-a9acce274458" (UID: "fdcec320-e05c-4739-abf1-a9acce274458"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:39:20.158759 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.158732    5423 generic.go:356] "Generic (PLEG): container finished" podID="fdcec320-e05c-4739-abf1-a9acce274458" containerID="293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d" exitCode=0
Nov 01 14:39:20.158871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.158777    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"
Nov 01 14:39:20.158871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.158818    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh" event={"ID":"fdcec320-e05c-4739-abf1-a9acce274458","Type":"ContainerDied","Data":"293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d"}
Nov 01 14:39:20.159014 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.158883    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh" event={"ID":"fdcec320-e05c-4739-abf1-a9acce274458","Type":"ContainerDied","Data":"851a2472d30faaaa2d7bb336d02fd8a61684ce50e4acfe7608c6c825cfc12817"}
Nov 01 14:39:20.159014 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.158943    5423 scope.go:122] "RemoveContainer" containerID="293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d"
Nov 01 14:39:20.170098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.170080    5423 scope.go:122] "RemoveContainer" containerID="293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d"
Nov 01 14:39:20.178428 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:39:20.173879    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d\": container with ID starting with 293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d not found: ID does not exist" containerID="293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d"
Nov 01 14:39:20.178428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.173944    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d"} err="failed to get container status \"293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d\": rpc error: code = NotFound desc = could not find container \"293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d\": container with ID starting with 293ffff85d1619437c5454aa9df70227a19da6b9d7b18c17709886f90e07080d not found: ID does not exist"
Nov 01 14:39:20.227875 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.227831    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9867\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-kube-api-access-x9867\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:39:20.227875 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.227876    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdcec320-e05c-4739-abf1-a9acce274458-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:39:20.282563 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.282530    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"]
Nov 01 14:39:20.286255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:20.286231    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dfd5db6bc-nhjdh"]
Nov 01 14:39:21.960923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:21.960887    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdcec320-e05c-4739-abf1-a9acce274458" path="/var/lib/kubelet/pods/fdcec320-e05c-4739-abf1-a9acce274458/volumes"
Nov 01 14:39:35.150785 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:39:35.150751    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:39:40.344819 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.344784    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"]
Nov 01 14:39:40.345437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.345422    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fdcec320-e05c-4739-abf1-a9acce274458" containerName="payload-processing"
Nov 01 14:39:40.345512 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.345437    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcec320-e05c-4739-abf1-a9acce274458" containerName="payload-processing"
Nov 01 14:39:40.345551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.345524    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcec320-e05c-4739-abf1-a9acce274458" containerName="payload-processing"
Nov 01 14:39:40.349041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.349026    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446081 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.445991    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446081 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.446049    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4c9db2-de3a-4021-8941-a2245eced8de-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446081 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.446068    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.446100    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.446157    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.446188    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.446237    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwvg\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-kube-api-access-jmwvg\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.446307    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.446457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.446376    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.452445 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.452415    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"]
Nov 01 14:39:40.467290 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.467260    5423 generic.go:356] "Generic (PLEG): container finished" podID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerID="b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c" exitCode=0
Nov 01 14:39:40.467415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.467331    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" event={"ID":"4fcb770b-5f04-44f0-9cae-d882ddd526a8","Type":"ContainerDied","Data":"b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c"}
Nov 01 14:39:40.467744 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.467732    5423 scope.go:122] "RemoveContainer" containerID="b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c"
Nov 01 14:39:40.469122 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.469092    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:40.469234 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.469174    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:40.469234 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.469222    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:40.547841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.547815    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.547958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.547876    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548001 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.547987    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548046 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548029    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4c9db2-de3a-4021-8941-a2245eced8de-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548160 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548138    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548222    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548227    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548362 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548276    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548362 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548350    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548362    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548406    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548476    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwvg\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-kube-api-access-jmwvg\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.548670 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548624    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.549109 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.548834    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4c9db2-de3a-4021-8941-a2245eced8de-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.550733 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.550714    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.550820 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.550751    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.638568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.638531    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.641471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.641328    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwvg\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-kube-api-access-jmwvg\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:40.662213 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:40.662004    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:41.096024 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:39:41.095991    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4c9db2_de3a_4021_8941_a2245eced8de.slice/crio-01a0ad79f2972077ede6b605403c071f5e645fbc29f3cc3aff790f10ac31b8b5 WatchSource:0}: Error finding container 01a0ad79f2972077ede6b605403c071f5e645fbc29f3cc3aff790f10ac31b8b5: Status 404 returned error can't find the container with id 01a0ad79f2972077ede6b605403c071f5e645fbc29f3cc3aff790f10ac31b8b5
Nov 01 14:39:41.099191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.099147    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:41.099311 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.099240    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:41.099311 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.099281    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:41.099455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.099429    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"]
Nov 01 14:39:41.476657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.476564    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" event={"ID":"4e4c9db2-de3a-4021-8941-a2245eced8de","Type":"ContainerStarted","Data":"1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a"}
Nov 01 14:39:41.476657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.476624    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" event={"ID":"4e4c9db2-de3a-4021-8941-a2245eced8de","Type":"ContainerStarted","Data":"01a0ad79f2972077ede6b605403c071f5e645fbc29f3cc3aff790f10ac31b8b5"}
Nov 01 14:39:41.478675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.478643    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" event={"ID":"4fcb770b-5f04-44f0-9cae-d882ddd526a8","Type":"ContainerStarted","Data":"6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf"}
Nov 01 14:39:41.603410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.596779    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" podStartSLOduration=1.5967610300000001 podStartE2EDuration="1.59676103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:41.592563841 +0000 UTC m=+1321.758049139" watchObservedRunningTime="2026-11-01 14:39:41.59676103 +0000 UTC m=+1321.762246329"
Nov 01 14:39:41.662691 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:41.662660    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:42.103012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:42.102941    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:39:42.180412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:42.177150    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:39:42.196174 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:42.192612    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:39:42.196174 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:42.192674    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 14:39:42.664745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:42.664713    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:39:42.665182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:42.664774    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 14:39:42.674864 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:42.674843    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:39:43.172609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:43.172550    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:39:43.172807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:43.172623    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 14:39:43.663350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:43.663316    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:39:43.663652 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:43.663362    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 14:39:44.183474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:44.183440    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:39:44.184015 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:44.183983    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 14:39:44.662809 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:44.662773    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:39:44.662988 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:44.662836    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 14:39:45.171324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:45.171278    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:39:45.171509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:45.171342    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 14:39:45.663217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:45.663183    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:39:45.663659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:45.663241    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 14:39:46.172240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:46.172212    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:39:46.172504 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:46.172478    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 14:39:46.663682 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:46.663635    5423 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:39:46.664248 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:46.663689    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 14:39:47.176376 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:47.176337    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:39:47.177668 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:47.177646    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:39:47.667698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:47.667667    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:47.668178 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:47.668068    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:47.668825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:47.668809    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:39:57.586665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.586626    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"]
Nov 01 14:39:57.592581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.592553    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.645202 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.645157    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a32dd3-openshift-default-dockercfg-prxb4\""
Nov 01 14:39:57.648274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648239    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-podinfo\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.648470 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648290    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-envoy\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.648470 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648352    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-credential-socket\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.648470 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648378    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktpp\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-kube-api-access-hktpp\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.648654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648525    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-data\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.648654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648625    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-socket\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.648825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648670    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-certs\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.648825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648728    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8aec01-28a3-4dba-81b8-c63eb0a32011-istiod-ca-cert\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.648825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.648767    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-token\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.700571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.700537    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"]
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.781635    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-podinfo\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.781689    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-envoy\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.781750    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-credential-socket\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.781776    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktpp\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-kube-api-access-hktpp\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.781856    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-data\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.781956    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-socket\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.782001    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-certs\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.782053    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8aec01-28a3-4dba-81b8-c63eb0a32011-istiod-ca-cert\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.782271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.782097    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-token\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.845647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.802860    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-certs\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.845647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.803190    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-socket\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.845647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.804075    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8aec01-28a3-4dba-81b8-c63eb0a32011-istiod-ca-cert\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.845647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.804312    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-credential-socket\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.846572 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.846544    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-data\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.862643 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.862605    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-envoy\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.863476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.863448    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-podinfo\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.865167 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.865143    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-token\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.883440 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.883412    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktpp\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-kube-api-access-hktpp\") pod \"e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") " pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:57.909450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:57.909422    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:58.348971 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:58.348929    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"]
Nov 01 14:39:58.387153 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:58.387035    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:58.387153 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:58.387112    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:58.387153 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:58.387147    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:39:58.705504 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:58.705382    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" event={"ID":"da8aec01-28a3-4dba-81b8-c63eb0a32011","Type":"ContainerStarted","Data":"13ce3ccdc16bee8e06940526b633636bd9a7ded3483f1a349acf354f7be8f8df"}
Nov 01 14:39:58.705504 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:58.705470    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" event={"ID":"da8aec01-28a3-4dba-81b8-c63eb0a32011","Type":"ContainerStarted","Data":"0924f216ac77fdab51d09ebe0ca6f1684dd63a18121dd1724adac7bab1cf19eb"}
Nov 01 14:39:58.826161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:58.826104    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" podStartSLOduration=1.826089873 podStartE2EDuration="1.826089873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:58.824265869 +0000 UTC m=+1338.989751167" watchObservedRunningTime="2026-11-01 14:39:58.826089873 +0000 UTC m=+1338.991575170"
Nov 01 14:39:58.909783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:58.909746    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:39:59.916021 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:39:59.915989    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:40:00.719411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:00.719339    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:40:00.720639 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:00.720607    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:40:05.149542 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:40:05.149505    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:40:05.196154 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.196119    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"]
Nov 01 14:40:05.201423 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.201377    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:05.300372 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.294513    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"]
Nov 01 14:40:05.325408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.325011    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a32dd3\""
Nov 01 14:40:05.325408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.325187    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a32dd3-dockercfg-w27xk\""
Nov 01 14:40:05.362622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.361630    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82907e08-246e-4dee-9b81-d99c7be37843-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59\" (UID: \"82907e08-246e-4dee-9b81-d99c7be37843\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:05.362622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.361690    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj478\" (UniqueName: \"kubernetes.io/projected/82907e08-246e-4dee-9b81-d99c7be37843-kube-api-access-gj478\") pod \"payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59\" (UID: \"82907e08-246e-4dee-9b81-d99c7be37843\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:05.381586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.381549    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"]
Nov 01 14:40:05.444491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.444415    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:05.464144 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.464112    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82907e08-246e-4dee-9b81-d99c7be37843-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59\" (UID: \"82907e08-246e-4dee-9b81-d99c7be37843\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:05.464370 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.464351    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj478\" (UniqueName: \"kubernetes.io/projected/82907e08-246e-4dee-9b81-d99c7be37843-kube-api-access-gj478\") pod \"payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59\" (UID: \"82907e08-246e-4dee-9b81-d99c7be37843\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:05.464914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.464888    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82907e08-246e-4dee-9b81-d99c7be37843-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59\" (UID: \"82907e08-246e-4dee-9b81-d99c7be37843\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:05.465007 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.464166    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"]
Nov 01 14:40:05.522377 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.522348    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj478\" (UniqueName: \"kubernetes.io/projected/82907e08-246e-4dee-9b81-d99c7be37843-kube-api-access-gj478\") pod \"payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59\" (UID: \"82907e08-246e-4dee-9b81-d99c7be37843\") " pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:05.565858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.565821    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:05.566298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.566271    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjb2l\" (UniqueName: \"kubernetes.io/projected/853493fd-adf5-4688-be1d-d267ce5523e5-kube-api-access-sjb2l\") pod \"payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z\" (UID: \"853493fd-adf5-4688-be1d-d267ce5523e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:05.566455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.566437    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853493fd-adf5-4688-be1d-d267ce5523e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z\" (UID: \"853493fd-adf5-4688-be1d-d267ce5523e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:05.668462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.668176    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjb2l\" (UniqueName: \"kubernetes.io/projected/853493fd-adf5-4688-be1d-d267ce5523e5-kube-api-access-sjb2l\") pod \"payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z\" (UID: \"853493fd-adf5-4688-be1d-d267ce5523e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:05.668462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.668306    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853493fd-adf5-4688-be1d-d267ce5523e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z\" (UID: \"853493fd-adf5-4688-be1d-d267ce5523e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:05.669776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.669746    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853493fd-adf5-4688-be1d-d267ce5523e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z\" (UID: \"853493fd-adf5-4688-be1d-d267ce5523e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:05.725904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.725827    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjb2l\" (UniqueName: \"kubernetes.io/projected/853493fd-adf5-4688-be1d-d267ce5523e5-kube-api-access-sjb2l\") pod \"payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z\" (UID: \"853493fd-adf5-4688-be1d-d267ce5523e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:05.797156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.797122    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:05.943849 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:40:05.943817    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82907e08_246e_4dee_9b81_d99c7be37843.slice/crio-74b009d17b41609cb9e8decc39b7e26a2214c1291f08768ec552ba5ab20735e2 WatchSource:0}: Error finding container 74b009d17b41609cb9e8decc39b7e26a2214c1291f08768ec552ba5ab20735e2: Status 404 returned error can't find the container with id 74b009d17b41609cb9e8decc39b7e26a2214c1291f08768ec552ba5ab20735e2
Nov 01 14:40:05.944939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:05.944911    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"]
Nov 01 14:40:06.167679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:40:06.167650    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853493fd_adf5_4688_be1d_d267ce5523e5.slice/crio-c4bd9672502c26522d1bcc42edce16b8c7ba1a56bffcd147b3eead97cb7bc2a7 WatchSource:0}: Error finding container c4bd9672502c26522d1bcc42edce16b8c7ba1a56bffcd147b3eead97cb7bc2a7: Status 404 returned error can't find the container with id c4bd9672502c26522d1bcc42edce16b8c7ba1a56bffcd147b3eead97cb7bc2a7
Nov 01 14:40:06.171201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:06.171179    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"]
Nov 01 14:40:06.863330 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:06.863290    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z" event={"ID":"853493fd-adf5-4688-be1d-d267ce5523e5","Type":"ContainerStarted","Data":"13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a"}
Nov 01 14:40:06.863554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:06.863356    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z" event={"ID":"853493fd-adf5-4688-be1d-d267ce5523e5","Type":"ContainerStarted","Data":"c4bd9672502c26522d1bcc42edce16b8c7ba1a56bffcd147b3eead97cb7bc2a7"}
Nov 01 14:40:06.863554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:06.863514    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:06.864947 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:06.864923    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59" event={"ID":"82907e08-246e-4dee-9b81-d99c7be37843","Type":"ContainerStarted","Data":"2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b"}
Nov 01 14:40:06.865100 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:06.864965    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59" event={"ID":"82907e08-246e-4dee-9b81-d99c7be37843","Type":"ContainerStarted","Data":"74b009d17b41609cb9e8decc39b7e26a2214c1291f08768ec552ba5ab20735e2"}
Nov 01 14:40:06.865100 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:06.865011    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:06.977152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:06.977101    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z" podStartSLOduration=1.977087609 podStartE2EDuration="1.977087609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:06.975429684 +0000 UTC m=+1347.140914983" watchObservedRunningTime="2026-11-01 14:40:06.977087609 +0000 UTC m=+1347.142572906"
Nov 01 14:40:07.099410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:07.099156    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59" podStartSLOduration=2.099131507 podStartE2EDuration="2.099131507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:07.087855616 +0000 UTC m=+1347.253340918" watchObservedRunningTime="2026-11-01 14:40:07.099131507 +0000 UTC m=+1347.264616806"
Nov 01 14:40:09.774022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:09.773985    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:40:17.875751 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:17.875722    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:40:17.876179 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:17.875796    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:40:28.683003 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.682898    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"]
Nov 01 14:40:28.692409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.688892    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.742469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.742437    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-116e85-openshift-default-dockercfg-9x49c\""
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783054    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-data\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783145    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bz6\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-kube-api-access-45bz6\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783297    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-credential-socket\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783374    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-socket\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783420    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-podinfo\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783466    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a3901c-5d4f-4378-9778-e4186b9863e3-istiod-ca-cert\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783501    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-certs\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783527    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-envoy\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.795404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.783607    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-token\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.801217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.801182    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"]
Nov 01 14:40:28.884833 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.884800    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bz6\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-kube-api-access-45bz6\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.884923    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-credential-socket\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.884975    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-socket\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885143    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-podinfo\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885210 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885175    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a3901c-5d4f-4378-9778-e4186b9863e3-istiod-ca-cert\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885210 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885199    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-certs\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885218    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-envoy\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885271    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-token\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885436 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885310    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-data\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885436 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885407    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-socket\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885639 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885599    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-certs\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885338    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-credential-socket\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885693    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-data\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.885927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.885910    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a3901c-5d4f-4378-9778-e4186b9863e3-istiod-ca-cert\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.887554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.887533    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-envoy\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.887867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.887847    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-podinfo\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.983466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.981808    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-token\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:28.991408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:28.987039    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bz6\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-kube-api-access-45bz6\") pod \"e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") " pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:29.057543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:29.057515    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:29.484605 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:40:29.484573    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a3901c_5d4f_4378_9778_e4186b9863e3.slice/crio-60391d929d70d07d20089060832e59e917e8c83fa71e6c1f1c1612590de2fb44 WatchSource:0}: Error finding container 60391d929d70d07d20089060832e59e917e8c83fa71e6c1f1c1612590de2fb44: Status 404 returned error can't find the container with id 60391d929d70d07d20089060832e59e917e8c83fa71e6c1f1c1612590de2fb44
Nov 01 14:40:29.486984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:29.486950    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:40:29.487082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:29.487017    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:40:29.487082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:29.487049    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:40:29.487589 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:29.487570    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"]
Nov 01 14:40:30.043566 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:30.043524    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" event={"ID":"55a3901c-5d4f-4378-9778-e4186b9863e3","Type":"ContainerStarted","Data":"25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e"}
Nov 01 14:40:30.043967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:30.043586    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" event={"ID":"55a3901c-5d4f-4378-9778-e4186b9863e3","Type":"ContainerStarted","Data":"60391d929d70d07d20089060832e59e917e8c83fa71e6c1f1c1612590de2fb44"}
Nov 01 14:40:30.058050 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:30.058029    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:30.063890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:30.063866    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:30.162466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:30.162398    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" podStartSLOduration=2.162369469 podStartE2EDuration="2.162369469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:30.159342866 +0000 UTC m=+1370.324828160" watchObservedRunningTime="2026-11-01 14:40:30.162369469 +0000 UTC m=+1370.327854819"
Nov 01 14:40:31.050970 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:31.050940    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:31.051903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:31.051884    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:40:35.149208 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:40:35.149169    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:40:36.444256 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.444207    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"]
Nov 01 14:40:36.450901 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.450874    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:36.490003 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.489969    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc292128-7480-4da8-b594-2f65538a5e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg\" (UID: \"bc292128-7480-4da8-b594-2f65538a5e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:36.490274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.490253    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j22rw\" (UniqueName: \"kubernetes.io/projected/bc292128-7480-4da8-b594-2f65538a5e5a-kube-api-access-j22rw\") pod \"payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg\" (UID: \"bc292128-7480-4da8-b594-2f65538a5e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:36.508996 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.508916    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"]
Nov 01 14:40:36.554150 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.547732    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-116e85-dockercfg-7ltss\""
Nov 01 14:40:36.554150 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.547829    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-116e85\""
Nov 01 14:40:36.601408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.598556    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc292128-7480-4da8-b594-2f65538a5e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg\" (UID: \"bc292128-7480-4da8-b594-2f65538a5e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:36.601408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.598697    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j22rw\" (UniqueName: \"kubernetes.io/projected/bc292128-7480-4da8-b594-2f65538a5e5a-kube-api-access-j22rw\") pod \"payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg\" (UID: \"bc292128-7480-4da8-b594-2f65538a5e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:36.707142 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.693688    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc292128-7480-4da8-b594-2f65538a5e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg\" (UID: \"bc292128-7480-4da8-b594-2f65538a5e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:36.816408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.815457    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j22rw\" (UniqueName: \"kubernetes.io/projected/bc292128-7480-4da8-b594-2f65538a5e5a-kube-api-access-j22rw\") pod \"payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg\" (UID: \"bc292128-7480-4da8-b594-2f65538a5e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:36.849153 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:36.824770    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:37.224066 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:40:37.224033    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc292128_7480_4da8_b594_2f65538a5e5a.slice/crio-bf41cd4c3b20fd861e13a2dc23b2298cf0ca6bf228b46ed0c20a4c791998ce3d WatchSource:0}: Error finding container bf41cd4c3b20fd861e13a2dc23b2298cf0ca6bf228b46ed0c20a4c791998ce3d: Status 404 returned error can't find the container with id bf41cd4c3b20fd861e13a2dc23b2298cf0ca6bf228b46ed0c20a4c791998ce3d
Nov 01 14:40:37.228430 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:37.228404    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"]
Nov 01 14:40:38.104359 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:38.104324    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg" event={"ID":"bc292128-7480-4da8-b594-2f65538a5e5a","Type":"ContainerStarted","Data":"2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c"}
Nov 01 14:40:38.104853 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:38.104379    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg" event={"ID":"bc292128-7480-4da8-b594-2f65538a5e5a","Type":"ContainerStarted","Data":"bf41cd4c3b20fd861e13a2dc23b2298cf0ca6bf228b46ed0c20a4c791998ce3d"}
Nov 01 14:40:38.104853 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:38.104608    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:40:38.225721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:38.225667    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg" podStartSLOduration=2.225653238 podStartE2EDuration="2.225653238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:38.222624489 +0000 UTC m=+1378.388109788" watchObservedRunningTime="2026-11-01 14:40:38.225653238 +0000 UTC m=+1378.391138535"
Nov 01 14:40:40.246354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:40.246324    5423 scope.go:122] "RemoveContainer" containerID="798fd4222f56b447909dc50defe09c3e7090753f34892740b53c6443cc9cdecc"
Nov 01 14:40:41.329906 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.329871    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"]
Nov 01 14:40:41.335797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.335773    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.394377 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.394333    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-credential-socket\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.410565 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.410528    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltt9\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-kube-api-access-9ltt9\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.412342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.412313    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-certs\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.412701 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.412682    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-socket\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.412990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.412937    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-podinfo\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.413177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.413146    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-token\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.413369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.413353    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-envoy\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.413708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.413679    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb43700-7fa7-457a-be62-bb3be96e9b10-istiod-ca-cert\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.414022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.414002    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-data\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.484671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.484629    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"]
Nov 01 14:40:41.516411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.485448    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0c4976-openshift-default-dockercfg-rnx7l\""
Nov 01 14:40:41.518962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.518028    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb43700-7fa7-457a-be62-bb3be96e9b10-istiod-ca-cert\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.534405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.519272    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-data\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.534737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.518845    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb43700-7fa7-457a-be62-bb3be96e9b10-istiod-ca-cert\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.534737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.530853    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-data\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.534737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.534716    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-credential-socket\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.534973 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.534774    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltt9\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-kube-api-access-9ltt9\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.534973 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.534841    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-certs\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.534973 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.534941    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-socket\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.535138 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.535022    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-podinfo\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.535138 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.535054    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-token\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.535138 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.535095    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-envoy\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.536952 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.535851    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-certs\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.536952 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.536207    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-credential-socket\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.547828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.547786    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-socket\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.565022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.564168    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-podinfo\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.569440 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.566140    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-envoy\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.791706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.786656    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-token\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:41.791706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:41.789070    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltt9\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-kube-api-access-9ltt9\") pod \"e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") " pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:42.010124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:42.009855    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:42.436447 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:40:42.436414    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb43700_7fa7_457a_be62_bb3be96e9b10.slice/crio-548bf7775bf170556f4ffbcee36ca0ab79cab0d4aacc6ad9be490982ba27ee13 WatchSource:0}: Error finding container 548bf7775bf170556f4ffbcee36ca0ab79cab0d4aacc6ad9be490982ba27ee13: Status 404 returned error can't find the container with id 548bf7775bf170556f4ffbcee36ca0ab79cab0d4aacc6ad9be490982ba27ee13
Nov 01 14:40:42.439433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:42.439365    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:40:42.439517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:42.439483    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:40:42.439560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:42.439532    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:40:42.439629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:42.439609    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"]
Nov 01 14:40:43.151980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:43.151936    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" event={"ID":"0bb43700-7fa7-457a-be62-bb3be96e9b10","Type":"ContainerStarted","Data":"12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea"}
Nov 01 14:40:43.152154 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:43.152020    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" event={"ID":"0bb43700-7fa7-457a-be62-bb3be96e9b10","Type":"ContainerStarted","Data":"548bf7775bf170556f4ffbcee36ca0ab79cab0d4aacc6ad9be490982ba27ee13"}
Nov 01 14:40:43.321690 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:43.321627    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" podStartSLOduration=2.321610778 podStartE2EDuration="2.321610778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:43.319755775 +0000 UTC m=+1383.485241073" watchObservedRunningTime="2026-11-01 14:40:43.321610778 +0000 UTC m=+1383.487096077"
Nov 01 14:40:44.010494 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:44.010455    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:44.016534 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:44.016506    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:44.158776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:44.158745    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:44.159697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:44.159675    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:40:45.561877 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:45.561802    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:40:49.115152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:40:49.115119    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:41:05.149345 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:41:05.149298    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:41:12.625724 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:12.625685    5423 generic.go:356] "Generic (PLEG): container finished" podID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerID="13ce3ccdc16bee8e06940526b633636bd9a7ded3483f1a349acf354f7be8f8df" exitCode=0
Nov 01 14:41:12.625724 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:12.625711    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" event={"ID":"da8aec01-28a3-4dba-81b8-c63eb0a32011","Type":"ContainerDied","Data":"13ce3ccdc16bee8e06940526b633636bd9a7ded3483f1a349acf354f7be8f8df"}
Nov 01 14:41:12.626201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:12.626080    5423 scope.go:122] "RemoveContainer" containerID="13ce3ccdc16bee8e06940526b633636bd9a7ded3483f1a349acf354f7be8f8df"
Nov 01 14:41:12.627547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:12.627522    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:41:12.627608 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:12.627586    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:41:12.627644 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:12.627618    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:41:13.635163 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:13.635122    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" event={"ID":"da8aec01-28a3-4dba-81b8-c63eb0a32011","Type":"ContainerStarted","Data":"e925c272c485cdf70171ec58c30cbee70a884c3360c8c5436117aa4c045ee39a"}
Nov 01 14:41:13.919565 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:13.919454    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:41:13.919720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:13.919613    5423 patch_prober.go:29] interesting pod/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 14:41:13.919720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:13.919699    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 14:41:14.919764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:14.919604    5423 patch_prober.go:29] interesting pod/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 14:41:14.920268 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:14.919794    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 14:41:15.719580 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.719534    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:41:15.787229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.787198    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7387-db8093"]
Nov 01 14:41:15.791512 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.791495    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:15.840853 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.840824    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-db8093"]
Nov 01 14:41:15.886161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.886118    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb84m\" (UniqueName: \"kubernetes.io/projected/85cae8df-f846-4583-94a5-63fb4f8630a0-kube-api-access-zb84m\") pod \"test-curl-7387-db8093\" (UID: \"85cae8df-f846-4583-94a5-63fb4f8630a0\") " pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:15.886716 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.886694    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jngw\""
Nov 01 14:41:15.919867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.919834    5423 patch_prober.go:29] interesting pod/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 14:41:15.920565 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.920522    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 14:41:15.987856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:15.987782    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb84m\" (UniqueName: \"kubernetes.io/projected/85cae8df-f846-4583-94a5-63fb4f8630a0-kube-api-access-zb84m\") pod \"test-curl-7387-db8093\" (UID: \"85cae8df-f846-4583-94a5-63fb4f8630a0\") " pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:16.044658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:16.044627    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb84m\" (UniqueName: \"kubernetes.io/projected/85cae8df-f846-4583-94a5-63fb4f8630a0-kube-api-access-zb84m\") pod \"test-curl-7387-db8093\" (UID: \"85cae8df-f846-4583-94a5-63fb4f8630a0\") " pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:16.128854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:16.128810    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:16.521725 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:41:16.521670    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85cae8df_f846_4583_94a5_63fb4f8630a0.slice/crio-8ff1010798e47c7d349b0dbc29bf525e676e245e5f9779c7c34b3f5b3b320adf WatchSource:0}: Error finding container 8ff1010798e47c7d349b0dbc29bf525e676e245e5f9779c7c34b3f5b3b320adf: Status 404 returned error can't find the container with id 8ff1010798e47c7d349b0dbc29bf525e676e245e5f9779c7c34b3f5b3b320adf
Nov 01 14:41:16.524449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:16.524422    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-db8093"]
Nov 01 14:41:16.655856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:16.655824    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-db8093" event={"ID":"85cae8df-f846-4583-94a5-63fb4f8630a0","Type":"ContainerStarted","Data":"8ff1010798e47c7d349b0dbc29bf525e676e245e5f9779c7c34b3f5b3b320adf"}
Nov 01 14:41:16.915514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:16.915474    5423 patch_prober.go:29] interesting pod/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 14:41:16.915784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:16.915751    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 14:41:17.056714 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:17.056678    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:41:17.916826 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:17.916785    5423 patch_prober.go:29] interesting pod/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 14:41:17.917117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:17.917084    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 14:41:18.910411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:18.910367    5423 patch_prober.go:29] interesting pod/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 14:41:18.910890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:18.910432    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 14:41:19.914332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:19.914307    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:41:19.915536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:19.915518    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:41:20.683606 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:20.683556    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-db8093" event={"ID":"85cae8df-f846-4583-94a5-63fb4f8630a0","Type":"ContainerStarted","Data":"425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80"}
Nov 01 14:41:20.799925 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:20.799871    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7387-db8093" podStartSLOduration=2.595909844 podStartE2EDuration="5.79985708s" totalImagesPullingTime="3.203947236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:41:20.796896965 +0000 UTC m=+1420.962382262" watchObservedRunningTime="2026-11-01 14:41:20.79985708 +0000 UTC m=+1420.965342379"
Nov 01 14:41:22.276933 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:22.276892    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7387-db8093"]
Nov 01 14:41:22.282134 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:22.282102    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7387-db8093"]
Nov 01 14:41:22.696376 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:22.696335    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7387-db8093" podUID="85cae8df-f846-4583-94a5-63fb4f8630a0" containerName="test-curl-7387-db8093" containerID="cri-o://425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80" gracePeriod=2
Nov 01 14:41:22.748154 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:41:22.748114    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-db8093\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="85cae8df-f846-4583-94a5-63fb4f8630a0" pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:24.943251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:24.943224    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:24.974749 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:24.974692    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cae8df-f846-4583-94a5-63fb4f8630a0-kube-api-access-zb84m\" (UniqueName: \"kubernetes.io/projected/85cae8df-f846-4583-94a5-63fb4f8630a0-kube-api-access-zb84m\") pod \"85cae8df-f846-4583-94a5-63fb4f8630a0\" (UID: \"85cae8df-f846-4583-94a5-63fb4f8630a0\") "
Nov 01 14:41:24.976886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:24.976859    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cae8df-f846-4583-94a5-63fb4f8630a0-kube-api-access-zb84m" pod "85cae8df-f846-4583-94a5-63fb4f8630a0" (UID: "85cae8df-f846-4583-94a5-63fb4f8630a0"). InnerVolumeSpecName "kube-api-access-zb84m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:41:25.076568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:25.076539    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb84m\" (UniqueName: \"kubernetes.io/projected/85cae8df-f846-4583-94a5-63fb4f8630a0-kube-api-access-zb84m\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:41:25.741482 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:25.741446    5423 generic.go:356] "Generic (PLEG): container finished" podID="85cae8df-f846-4583-94a5-63fb4f8630a0" containerID="425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80" exitCode=137
Nov 01 14:41:25.741675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:25.741499    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:25.741675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:25.741523    5423 scope.go:122] "RemoveContainer" containerID="425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80"
Nov 01 14:41:25.750947 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:25.750920    5423 scope.go:122] "RemoveContainer" containerID="425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80"
Nov 01 14:41:25.751207 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:41:25.751187    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80\": container with ID starting with 425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80 not found: ID does not exist" containerID="425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80"
Nov 01 14:41:25.751247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:25.751215    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80"} err="failed to get container status \"425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80\": rpc error: code = NotFound desc = could not find container \"425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80\": container with ID starting with 425fc443cebe62af5da00e06de60d4a5137cfc0200b9f12b845e14ec91c36e80 not found: ID does not exist"
Nov 01 14:41:25.801482 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:41:25.801457    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-db8093\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="85cae8df-f846-4583-94a5-63fb4f8630a0" pod="opendatahub/test-curl-7387-db8093"
Nov 01 14:41:25.962802 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:25.962767    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85cae8df-f846-4583-94a5-63fb4f8630a0" path="/var/lib/kubelet/pods/85cae8df-f846-4583-94a5-63fb4f8630a0/volumes"
Nov 01 14:41:34.589481 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:34.589440    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"]
Nov 01 14:41:35.253457 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:41:35.249464    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:41:35.274506 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.274473    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"]
Nov 01 14:41:35.275769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.275739    5423 cpu_manager.go:414] "containerMap: removing container" podUID="85cae8df-f846-4583-94a5-63fb4f8630a0" containerName="test-curl-7387-db8093"
Nov 01 14:41:35.275937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.275919    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cae8df-f846-4583-94a5-63fb4f8630a0" containerName="test-curl-7387-db8093"
Nov 01 14:41:35.276258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.276240    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cae8df-f846-4583-94a5-63fb4f8630a0" containerName="test-curl-7387-db8093"
Nov 01 14:41:35.323419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.323351    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"]
Nov 01 14:41:35.323609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.323549    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.376525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.376485    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0c4976-serving-cert\""
Nov 01 14:41:35.454524 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.453979    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800238c0-bf47-460e-b0ca-18fba0a9a120-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.454524 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.454101    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtpvw\" (UniqueName: \"kubernetes.io/projected/800238c0-bf47-460e-b0ca-18fba0a9a120-kube-api-access-gtpvw\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.454524 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.454233    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-maas-api-tls\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.454524 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.454323    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-metrics-certs\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.556117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.556090    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtpvw\" (UniqueName: \"kubernetes.io/projected/800238c0-bf47-460e-b0ca-18fba0a9a120-kube-api-access-gtpvw\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.556290 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.556207    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-maas-api-tls\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.556290 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.556276    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-metrics-certs\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.556450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.556411    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800238c0-bf47-460e-b0ca-18fba0a9a120-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.557108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.557077    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800238c0-bf47-460e-b0ca-18fba0a9a120-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.558791 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.558760    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-maas-api-tls\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.559013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.558994    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-metrics-certs\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.661190 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.661154    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtpvw\" (UniqueName: \"kubernetes.io/projected/800238c0-bf47-460e-b0ca-18fba0a9a120-kube-api-access-gtpvw\") pod \"maas-api-e2e-api-a-0c4976-76f69455f8-v5b77\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:35.944440 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:35.944328    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:36.953067 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:41:36.953018    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800238c0_bf47_460e_b0ca_18fba0a9a120.slice/crio-bda070f752cc8a29e5aad2c0e835a4bcc0c86b53bc68e293da5adb9c55a06f3d WatchSource:0}: Error finding container bda070f752cc8a29e5aad2c0e835a4bcc0c86b53bc68e293da5adb9c55a06f3d: Status 404 returned error can't find the container with id bda070f752cc8a29e5aad2c0e835a4bcc0c86b53bc68e293da5adb9c55a06f3d
Nov 01 14:41:36.957526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:36.957502    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"]
Nov 01 14:41:37.021127 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:37.021093    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77" event={"ID":"800238c0-bf47-460e-b0ca-18fba0a9a120","Type":"ContainerStarted","Data":"bda070f752cc8a29e5aad2c0e835a4bcc0c86b53bc68e293da5adb9c55a06f3d"}
Nov 01 14:41:39.049551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:39.049506    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77" event={"ID":"800238c0-bf47-460e-b0ca-18fba0a9a120","Type":"ContainerStarted","Data":"75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a"}
Nov 01 14:41:39.049942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:39.049604    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:39.181401 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:39.181347    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77" podStartSLOduration=3.191716404 podStartE2EDuration="4.181330591s" totalImagesPullingTime="989.614187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:41:39.180442274 +0000 UTC m=+1439.345927566" watchObservedRunningTime="2026-11-01 14:41:39.181330591 +0000 UTC m=+1439.346815891"
Nov 01 14:41:39.590339 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:39.590304    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd" podUID="1c02c67d-5add-4ff3-801a-295885e693b9" containerName="payload-pre-processing" containerID="cri-o://14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623" gracePeriod=25
Nov 01 14:41:45.104412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:45.099766    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:41:46.052236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.052135    5423 patch_prober.go:29] interesting pod/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 14:41:46.052495 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.052223    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 14:41:46.143292 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.143262    5423 generic.go:356] "Generic (PLEG): container finished" podID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerID="25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e" exitCode=0
Nov 01 14:41:46.143684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.143340    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" event={"ID":"55a3901c-5d4f-4378-9778-e4186b9863e3","Type":"ContainerDied","Data":"25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e"}
Nov 01 14:41:46.143794 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.143779    5423 scope.go:122] "RemoveContainer" containerID="25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e"
Nov 01 14:41:46.145316 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.145289    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:41:46.145369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.145357    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:41:46.145433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.145405    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:41:46.501271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:46.501226    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd" podUID="1c02c67d-5add-4ff3-801a-295885e693b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:41:47.151611 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:47.151578    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" event={"ID":"55a3901c-5d4f-4378-9778-e4186b9863e3","Type":"ContainerStarted","Data":"85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5"}
Nov 01 14:41:48.058784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:48.058751    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:41:48.058964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:48.058930    5423 patch_prober.go:29] interesting pod/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 14:41:48.059018 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:48.058986    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 14:41:49.058531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:49.058497    5423 patch_prober.go:29] interesting pod/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 14:41:49.058914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:49.058560    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 14:41:50.058261 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:50.058224    5423 patch_prober.go:29] interesting pod/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 14:41:50.058486 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:50.058280    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 14:41:51.058009 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:51.057965    5423 patch_prober.go:29] interesting pod/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 14:41:51.058430 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:51.058024    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 14:41:52.058771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:52.058735    5423 patch_prober.go:29] interesting pod/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 01 14:41:52.059179 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:52.058788    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 01 14:41:53.062908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:53.062881    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:41:53.063310 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:53.063293    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:41:53.063923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:53.063903    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:41:56.500764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:56.500720    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd" podUID="1c02c67d-5add-4ff3-801a-295885e693b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:41:57.807061 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:57.807021    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"]
Nov 01 14:41:59.869123 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:59.869068    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:41:59.969629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:59.969599    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c02c67d-5add-4ff3-801a-295885e693b9-kube-api-access-pnlcx\" (UniqueName: \"kubernetes.io/projected/1c02c67d-5add-4ff3-801a-295885e693b9-kube-api-access-pnlcx\") pod \"1c02c67d-5add-4ff3-801a-295885e693b9\" (UID: \"1c02c67d-5add-4ff3-801a-295885e693b9\") "
Nov 01 14:41:59.969799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:59.969696    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c02c67d-5add-4ff3-801a-295885e693b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c02c67d-5add-4ff3-801a-295885e693b9-plugins-config-volume\") pod \"1c02c67d-5add-4ff3-801a-295885e693b9\" (UID: \"1c02c67d-5add-4ff3-801a-295885e693b9\") "
Nov 01 14:41:59.970065 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:59.970043    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c02c67d-5add-4ff3-801a-295885e693b9-plugins-config-volume" pod "1c02c67d-5add-4ff3-801a-295885e693b9" (UID: "1c02c67d-5add-4ff3-801a-295885e693b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:41:59.970324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:59.970310    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c02c67d-5add-4ff3-801a-295885e693b9-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:41:59.971723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:41:59.971688    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c02c67d-5add-4ff3-801a-295885e693b9-kube-api-access-pnlcx" pod "1c02c67d-5add-4ff3-801a-295885e693b9" (UID: "1c02c67d-5add-4ff3-801a-295885e693b9"). InnerVolumeSpecName "kube-api-access-pnlcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:00.072062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.072025    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnlcx\" (UniqueName: \"kubernetes.io/projected/1c02c67d-5add-4ff3-801a-295885e693b9-kube-api-access-pnlcx\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:00.289498 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.289379    5423 generic.go:356] "Generic (PLEG): container finished" podID="1c02c67d-5add-4ff3-801a-295885e693b9" containerID="14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623" exitCode=0
Nov 01 14:42:00.289830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.289807    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd" event={"ID":"1c02c67d-5add-4ff3-801a-295885e693b9","Type":"ContainerDied","Data":"14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623"}
Nov 01 14:42:00.289980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.289962    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd" event={"ID":"1c02c67d-5add-4ff3-801a-295885e693b9","Type":"ContainerDied","Data":"ad43c1dbab03af746a94117574a9f6dc172827e5c12eb2d1b9428cf11d84a083"}
Nov 01 14:42:00.290094 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.290081    5423 scope.go:122] "RemoveContainer" containerID="14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623"
Nov 01 14:42:00.290356 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.290337    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"
Nov 01 14:42:00.307521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.307494    5423 scope.go:122] "RemoveContainer" containerID="14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623"
Nov 01 14:42:00.307848 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:00.307825    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623\": container with ID starting with 14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623 not found: ID does not exist" containerID="14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623"
Nov 01 14:42:00.307936 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.307859    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623"} err="failed to get container status \"14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623\": rpc error: code = NotFound desc = could not find container \"14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623\": container with ID starting with 14f41624b07ecaccfc941e822afac03480092c55540f1cdb8257d6f11805c623 not found: ID does not exist"
Nov 01 14:42:00.459862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.459827    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"]
Nov 01 14:42:00.511100 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:00.511068    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-2v2pd"]
Nov 01 14:42:01.401599 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:01.401561    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:42:01.962920 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:01.962887    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c02c67d-5add-4ff3-801a-295885e693b9" path="/var/lib/kubelet/pods/1c02c67d-5add-4ff3-801a-295885e693b9/volumes"
Nov 01 14:42:02.808407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:02.808342    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp" podUID="6f8385d2-567e-4897-a1e7-758dc183e95b" containerName="payload-processing" containerID="cri-o://c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4" gracePeriod=25
Nov 01 14:42:03.082082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.082058    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:42:03.246069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.246037    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8385d2-567e-4897-a1e7-758dc183e95b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8385d2-567e-4897-a1e7-758dc183e95b-plugins-config-volume\") pod \"6f8385d2-567e-4897-a1e7-758dc183e95b\" (UID: \"6f8385d2-567e-4897-a1e7-758dc183e95b\") "
Nov 01 14:42:03.246257 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.246236    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8385d2-567e-4897-a1e7-758dc183e95b-kube-api-access-zwpzj\" (UniqueName: \"kubernetes.io/projected/6f8385d2-567e-4897-a1e7-758dc183e95b-kube-api-access-zwpzj\") pod \"6f8385d2-567e-4897-a1e7-758dc183e95b\" (UID: \"6f8385d2-567e-4897-a1e7-758dc183e95b\") "
Nov 01 14:42:03.246506 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.246478    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8385d2-567e-4897-a1e7-758dc183e95b-plugins-config-volume" pod "6f8385d2-567e-4897-a1e7-758dc183e95b" (UID: "6f8385d2-567e-4897-a1e7-758dc183e95b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:03.246886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.246861    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8385d2-567e-4897-a1e7-758dc183e95b-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:03.248502 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.248477    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8385d2-567e-4897-a1e7-758dc183e95b-kube-api-access-zwpzj" pod "6f8385d2-567e-4897-a1e7-758dc183e95b" (UID: "6f8385d2-567e-4897-a1e7-758dc183e95b"). InnerVolumeSpecName "kube-api-access-zwpzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:03.349032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.348946    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwpzj\" (UniqueName: \"kubernetes.io/projected/6f8385d2-567e-4897-a1e7-758dc183e95b-kube-api-access-zwpzj\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:03.362224 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.362188    5423 generic.go:356] "Generic (PLEG): container finished" podID="6f8385d2-567e-4897-a1e7-758dc183e95b" containerID="c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4" exitCode=0
Nov 01 14:42:03.362421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.362233    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"
Nov 01 14:42:03.362421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.362257    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp" event={"ID":"6f8385d2-567e-4897-a1e7-758dc183e95b","Type":"ContainerDied","Data":"c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4"}
Nov 01 14:42:03.362421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.362314    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp" event={"ID":"6f8385d2-567e-4897-a1e7-758dc183e95b","Type":"ContainerDied","Data":"0176d967effba21e60c50f78a2b607b5ecf5c62898bae01f9b2562d5cfc8a0c9"}
Nov 01 14:42:03.362421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.362343    5423 scope.go:122] "RemoveContainer" containerID="c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4"
Nov 01 14:42:03.373352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.373323    5423 scope.go:122] "RemoveContainer" containerID="c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4"
Nov 01 14:42:03.373669 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:03.373640    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4\": container with ID starting with c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4 not found: ID does not exist" containerID="c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4"
Nov 01 14:42:03.373757 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.373680    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4"} err="failed to get container status \"c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4\": rpc error: code = NotFound desc = could not find container \"c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4\": container with ID starting with c8f7cc25ec299f3631b8655f1cad5d9922967e4e5b3944301cb529785315c6e4 not found: ID does not exist"
Nov 01 14:42:03.528827 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.528783    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"]
Nov 01 14:42:03.534499 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.534477    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84ded4-59b456d4f-9twjp"]
Nov 01 14:42:03.962367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:03.962335    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8385d2-567e-4897-a1e7-758dc183e95b" path="/var/lib/kubelet/pods/6f8385d2-567e-4897-a1e7-758dc183e95b/volumes"
Nov 01 14:42:05.155567 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:05.155532    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:42:12.792201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:12.792168    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"]
Nov 01 14:42:12.792729 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:12.792403    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n" podUID="4a6b8eb1-df62-4619-b07b-97a16b56afba" containerName="maas-api" containerID="cri-o://150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207" gracePeriod=30
Nov 01 14:42:13.190765 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.187588    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:42:13.266417 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.249167    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6b8eb1-df62-4619-b07b-97a16b56afba-kube-api-access-5jhd4\" (UniqueName: \"kubernetes.io/projected/4a6b8eb1-df62-4619-b07b-97a16b56afba-kube-api-access-5jhd4\") pod \"4a6b8eb1-df62-4619-b07b-97a16b56afba\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") "
Nov 01 14:42:13.266417 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.265852    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-metrics-certs\") pod \"4a6b8eb1-df62-4619-b07b-97a16b56afba\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") "
Nov 01 14:42:13.266417 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.265896    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls\") pod \"4a6b8eb1-df62-4619-b07b-97a16b56afba\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") "
Nov 01 14:42:13.266757 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.266371    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6b8eb1-df62-4619-b07b-97a16b56afba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6b8eb1-df62-4619-b07b-97a16b56afba-openshift-service-ca\") pod \"4a6b8eb1-df62-4619-b07b-97a16b56afba\" (UID: \"4a6b8eb1-df62-4619-b07b-97a16b56afba\") "
Nov 01 14:42:13.267016 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.266727    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6b8eb1-df62-4619-b07b-97a16b56afba-openshift-service-ca" pod "4a6b8eb1-df62-4619-b07b-97a16b56afba" (UID: "4a6b8eb1-df62-4619-b07b-97a16b56afba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:13.271766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.267701    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6b8eb1-df62-4619-b07b-97a16b56afba-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:13.271766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.271056    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls" pod "4a6b8eb1-df62-4619-b07b-97a16b56afba" (UID: "4a6b8eb1-df62-4619-b07b-97a16b56afba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:42:13.271766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.271716    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-metrics-certs" pod "4a6b8eb1-df62-4619-b07b-97a16b56afba" (UID: "4a6b8eb1-df62-4619-b07b-97a16b56afba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:42:13.275557 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.275526    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6b8eb1-df62-4619-b07b-97a16b56afba-kube-api-access-5jhd4" pod "4a6b8eb1-df62-4619-b07b-97a16b56afba" (UID: "4a6b8eb1-df62-4619-b07b-97a16b56afba"). InnerVolumeSpecName "kube-api-access-5jhd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:13.381817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.379829    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jhd4\" (UniqueName: \"kubernetes.io/projected/4a6b8eb1-df62-4619-b07b-97a16b56afba-kube-api-access-5jhd4\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:13.381817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.379878    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:13.381817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.379893    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6b8eb1-df62-4619-b07b-97a16b56afba-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:13.433304 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.433266    5423 generic.go:356] "Generic (PLEG): container finished" podID="4a6b8eb1-df62-4619-b07b-97a16b56afba" containerID="150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207" exitCode=0
Nov 01 14:42:13.433496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.433336    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"
Nov 01 14:42:13.433496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.433359    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n" event={"ID":"4a6b8eb1-df62-4619-b07b-97a16b56afba","Type":"ContainerDied","Data":"150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207"}
Nov 01 14:42:13.433590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.433499    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n" event={"ID":"4a6b8eb1-df62-4619-b07b-97a16b56afba","Type":"ContainerDied","Data":"bcc31bba1a4aee315c997f40c792d8550de2453b0723bde8e6f27e3adad38e28"}
Nov 01 14:42:13.433590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.433538    5423 scope.go:122] "RemoveContainer" containerID="150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207"
Nov 01 14:42:13.447182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.447165    5423 scope.go:122] "RemoveContainer" containerID="150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207"
Nov 01 14:42:13.447464 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:13.447446    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207\": container with ID starting with 150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207 not found: ID does not exist" containerID="150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207"
Nov 01 14:42:13.447536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.447471    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207"} err="failed to get container status \"150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207\": rpc error: code = NotFound desc = could not find container \"150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207\": container with ID starting with 150545fd621c1313e6daf46d02a165d52991c08a9bf9d2a14dea56c51d447207 not found: ID does not exist"
Nov 01 14:42:13.601546 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.601515    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"]
Nov 01 14:42:13.605680 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.605651    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84ded4-768f6567c4-x5k6n"]
Nov 01 14:42:13.996410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:13.993608    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6b8eb1-df62-4619-b07b-97a16b56afba" path="/var/lib/kubelet/pods/4a6b8eb1-df62-4619-b07b-97a16b56afba/volumes"
Nov 01 14:42:15.126656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.126621    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"]
Nov 01 14:42:15.140247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140205    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6f8385d2-567e-4897-a1e7-758dc183e95b" containerName="payload-processing"
Nov 01 14:42:15.140247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140240    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8385d2-567e-4897-a1e7-758dc183e95b" containerName="payload-processing"
Nov 01 14:42:15.140543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140303    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1c02c67d-5add-4ff3-801a-295885e693b9" containerName="payload-pre-processing"
Nov 01 14:42:15.140543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140311    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c02c67d-5add-4ff3-801a-295885e693b9" containerName="payload-pre-processing"
Nov 01 14:42:15.140543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140326    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4a6b8eb1-df62-4619-b07b-97a16b56afba" containerName="maas-api"
Nov 01 14:42:15.140543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140334    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6b8eb1-df62-4619-b07b-97a16b56afba" containerName="maas-api"
Nov 01 14:42:15.140869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140549    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c02c67d-5add-4ff3-801a-295885e693b9" containerName="payload-pre-processing"
Nov 01 14:42:15.140869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140568    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6b8eb1-df62-4619-b07b-97a16b56afba" containerName="maas-api"
Nov 01 14:42:15.140869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.140585    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8385d2-567e-4897-a1e7-758dc183e95b" containerName="payload-processing"
Nov 01 14:42:15.146419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.146317    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:15.179231 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.179200    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"]
Nov 01 14:42:15.263904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.263867    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-18a879-dockercfg-gk6zb\""
Nov 01 14:42:15.278464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.264759    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-18a879\""
Nov 01 14:42:15.305642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.280715    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7599a1-3847-46aa-874d-4a738de877e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r\" (UID: \"ac7599a1-3847-46aa-874d-4a738de877e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:15.306139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.306109    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2pzt\" (UniqueName: \"kubernetes.io/projected/ac7599a1-3847-46aa-874d-4a738de877e0-kube-api-access-f2pzt\") pod \"payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r\" (UID: \"ac7599a1-3847-46aa-874d-4a738de877e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:15.396463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.395443    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"]
Nov 01 14:42:15.415332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.415294    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2pzt\" (UniqueName: \"kubernetes.io/projected/ac7599a1-3847-46aa-874d-4a738de877e0-kube-api-access-f2pzt\") pod \"payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r\" (UID: \"ac7599a1-3847-46aa-874d-4a738de877e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:15.443828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.416145    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7599a1-3847-46aa-874d-4a738de877e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r\" (UID: \"ac7599a1-3847-46aa-874d-4a738de877e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:15.443828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.442696    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:15.443828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.442600    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"]
Nov 01 14:42:15.443828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.422896    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7599a1-3847-46aa-874d-4a738de877e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r\" (UID: \"ac7599a1-3847-46aa-874d-4a738de877e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:15.473808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.473779    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2pzt\" (UniqueName: \"kubernetes.io/projected/ac7599a1-3847-46aa-874d-4a738de877e0-kube-api-access-f2pzt\") pod \"payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r\" (UID: \"ac7599a1-3847-46aa-874d-4a738de877e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:15.492569 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.492539    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:15.544872 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.544785    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59\" (UID: \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:15.545020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.544930    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzq47\" (UniqueName: \"kubernetes.io/projected/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-kube-api-access-qzq47\") pod \"payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59\" (UID: \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:15.646870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.646780    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59\" (UID: \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:15.647029 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.646870    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzq47\" (UniqueName: \"kubernetes.io/projected/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-kube-api-access-qzq47\") pod \"payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59\" (UID: \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:15.647577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.647554    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59\" (UID: \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:15.704717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.704687    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzq47\" (UniqueName: \"kubernetes.io/projected/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-kube-api-access-qzq47\") pod \"payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59\" (UID: \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:15.796785 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.796755    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:15.872472 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:42:15.867485    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7599a1_3847_46aa_874d_4a738de877e0.slice/crio-d56d507b31e2fb0d007908ab1297b73f166da42308291699c8a718b30afb0b65 WatchSource:0}: Error finding container d56d507b31e2fb0d007908ab1297b73f166da42308291699c8a718b30afb0b65: Status 404 returned error can't find the container with id d56d507b31e2fb0d007908ab1297b73f166da42308291699c8a718b30afb0b65
Nov 01 14:42:15.872723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:15.872516    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"]
Nov 01 14:42:16.169763 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:42:16.169731    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a04f92_ec68_40e5_9a2b_39bdbf4919be.slice/crio-04eff6c78eb557f73d59c745088f96031f22df81c3c4ec08a8ea4f6f2d824da4 WatchSource:0}: Error finding container 04eff6c78eb557f73d59c745088f96031f22df81c3c4ec08a8ea4f6f2d824da4: Status 404 returned error can't find the container with id 04eff6c78eb557f73d59c745088f96031f22df81c3c4ec08a8ea4f6f2d824da4
Nov 01 14:42:16.172646 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.172575    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"]
Nov 01 14:42:16.478881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.478797    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59" event={"ID":"e9a04f92-ec68-40e5-9a2b-39bdbf4919be","Type":"ContainerStarted","Data":"f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7"}
Nov 01 14:42:16.478881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.478849    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59" event={"ID":"e9a04f92-ec68-40e5-9a2b-39bdbf4919be","Type":"ContainerStarted","Data":"04eff6c78eb557f73d59c745088f96031f22df81c3c4ec08a8ea4f6f2d824da4"}
Nov 01 14:42:16.479168 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.478917    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:16.480516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.480492    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r" event={"ID":"ac7599a1-3847-46aa-874d-4a738de877e0","Type":"ContainerStarted","Data":"f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e"}
Nov 01 14:42:16.480625 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.480528    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r" event={"ID":"ac7599a1-3847-46aa-874d-4a738de877e0","Type":"ContainerStarted","Data":"d56d507b31e2fb0d007908ab1297b73f166da42308291699c8a718b30afb0b65"}
Nov 01 14:42:16.480664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.480628    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:16.595090 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.595046    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59" podStartSLOduration=1.595034136 podStartE2EDuration="1.595034136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:42:16.593821502 +0000 UTC m=+1476.759306797" watchObservedRunningTime="2026-11-01 14:42:16.595034136 +0000 UTC m=+1476.760519489"
Nov 01 14:42:16.728409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:16.726942    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r" podStartSLOduration=1.726913405 podStartE2EDuration="1.726913405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:42:16.719614344 +0000 UTC m=+1476.885099643" watchObservedRunningTime="2026-11-01 14:42:16.726913405 +0000 UTC m=+1476.892398704"
Nov 01 14:42:19.142409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:19.139803    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"]
Nov 01 14:42:19.142409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:19.141009    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy" containerID="cri-o://1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a" gracePeriod=30
Nov 01 14:42:19.167674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:19.162423    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"]
Nov 01 14:42:19.167674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:19.162672    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy" containerID="cri-o://6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf" gracePeriod=30
Nov 01 14:42:24.436902 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.436874    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:42:24.501437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501380    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-data\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.501579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501484    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-kube-api-access-jmwvg\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-kube-api-access-jmwvg\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.501579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501538    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-credential-socket\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.501696 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501614    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-data" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.501696 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501645    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4c9db2-de3a-4021-8941-a2245eced8de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4c9db2-de3a-4021-8941-a2245eced8de-istiod-ca-cert\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.501696 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501691    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-podinfo\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.501855 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501806    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-credential-socket" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.501855 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501829    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-envoy\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.501944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501936    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-token\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-token\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.501981 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501960    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-certs\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.502034 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.501988    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4c9db2-de3a-4021-8941-a2245eced8de-istiod-ca-cert" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:24.502034 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.502022    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-socket\") pod \"4e4c9db2-de3a-4021-8941-a2245eced8de\" (UID: \"4e4c9db2-de3a-4021-8941-a2245eced8de\") "
Nov 01 14:42:24.502297 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.502272    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-certs" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.502507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.502475    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-socket" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.502835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.502808    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.502958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.502841    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4c9db2-de3a-4021-8941-a2245eced8de-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.502958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.502857    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.502958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.502872    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.502958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.502887    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.503762 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.503739    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-kube-api-access-jmwvg" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "kube-api-access-jmwvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:24.503896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.503879    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-podinfo" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:42:24.504088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.504065    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-envoy" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.504088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.504074    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-token" pod "4e4c9db2-de3a-4021-8941-a2245eced8de" (UID: "4e4c9db2-de3a-4021-8941-a2245eced8de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:24.555217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.555184    5423 generic.go:356] "Generic (PLEG): container finished" podID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerID="1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a" exitCode=0
Nov 01 14:42:24.555359 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.555246    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"
Nov 01 14:42:24.555359 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.555247    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" event={"ID":"4e4c9db2-de3a-4021-8941-a2245eced8de","Type":"ContainerDied","Data":"1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a"}
Nov 01 14:42:24.555524 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.555417    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x" event={"ID":"4e4c9db2-de3a-4021-8941-a2245eced8de","Type":"ContainerDied","Data":"01a0ad79f2972077ede6b605403c071f5e645fbc29f3cc3aff790f10ac31b8b5"}
Nov 01 14:42:24.555524 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.555436    5423 scope.go:122] "RemoveContainer" containerID="1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a"
Nov 01 14:42:24.604308 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.604147    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.604308 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.604185    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.604308 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.604200    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.604308 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.604215    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmwvg\" (UniqueName: \"kubernetes.io/projected/4e4c9db2-de3a-4021-8941-a2245eced8de-kube-api-access-jmwvg\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.637036 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.637017    5423 scope.go:122] "RemoveContainer" containerID="1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a"
Nov 01 14:42:24.637371 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:24.637348    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a\": container with ID starting with 1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a not found: ID does not exist" containerID="1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a"
Nov 01 14:42:24.637515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.637433    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a"} err="failed to get container status \"1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a\": rpc error: code = NotFound desc = could not find container \"1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a\": container with ID starting with 1c841f0f3f3c6d2c7dfe6efc7283e7fb38295befda865c5e425fdf86b78f7f1a not found: ID does not exist"
Nov 01 14:42:24.718169 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.718134    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"]
Nov 01 14:42:24.723582 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.723553    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wrn2x"]
Nov 01 14:42:24.785243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.785205    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:42:24.909201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909169    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-socket\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.909432 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909212    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-data\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.909432 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909292    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istiod-ca-cert\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.909432 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909425    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-podinfo\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.909615 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909460    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-certs\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.909615 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909504    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-socket" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.909615 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909583    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-data" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.909825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909515    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-token\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-token\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.909825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909708    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-envoy\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.909825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909748    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-kube-api-access-c697p\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-kube-api-access-c697p\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.909962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909846    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istiod-ca-cert" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:24.909962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.909927    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-credential-socket\") pod \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\" (UID: \"4fcb770b-5f04-44f0-9cae-d882ddd526a8\") "
Nov 01 14:42:24.910449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.910414    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-certs" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.910697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.910600    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.910697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.910622    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-credential-socket" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.910697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.910630    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.910697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.910661    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.910697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.910675    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.912580 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.912541    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-envoy" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.913381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.912880    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-podinfo" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:42:24.914059 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.913952    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-kube-api-access-c697p" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "kube-api-access-c697p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:24.915658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:24.915631    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-token" pod "4fcb770b-5f04-44f0-9cae-d882ddd526a8" (UID: "4fcb770b-5f04-44f0-9cae-d882ddd526a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:25.011940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.011858    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:25.011940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.011887    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:25.011940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.011897    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:25.011940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.011906    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c697p\" (UniqueName: \"kubernetes.io/projected/4fcb770b-5f04-44f0-9cae-d882ddd526a8-kube-api-access-c697p\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:25.011940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.011916    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fcb770b-5f04-44f0-9cae-d882ddd526a8-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:25.564842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.564807    5423 generic.go:356] "Generic (PLEG): container finished" podID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerID="6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf" exitCode=0
Nov 01 14:42:25.565240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.564863    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"
Nov 01 14:42:25.565240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.564884    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" event={"ID":"4fcb770b-5f04-44f0-9cae-d882ddd526a8","Type":"ContainerDied","Data":"6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf"}
Nov 01 14:42:25.565240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.564933    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4" event={"ID":"4fcb770b-5f04-44f0-9cae-d882ddd526a8","Type":"ContainerDied","Data":"eb5220112ae19eca93440809281074efddf592262fae2b522e5871c63661dde0"}
Nov 01 14:42:25.565240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.564955    5423 scope.go:122] "RemoveContainer" containerID="6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf"
Nov 01 14:42:25.575479 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.575457    5423 scope.go:122] "RemoveContainer" containerID="b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c"
Nov 01 14:42:25.585332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.585311    5423 scope.go:122] "RemoveContainer" containerID="6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf"
Nov 01 14:42:25.585647 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:25.585628    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf\": container with ID starting with 6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf not found: ID does not exist" containerID="6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf"
Nov 01 14:42:25.585732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.585655    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf"} err="failed to get container status \"6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf\": rpc error: code = NotFound desc = could not find container \"6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf\": container with ID starting with 6d806a6a29cd05b53d9f2292834c352f6d51a40aa537edf15c326401ce52f8bf not found: ID does not exist"
Nov 01 14:42:25.585732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.585673    5423 scope.go:122] "RemoveContainer" containerID="b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c"
Nov 01 14:42:25.585921 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:25.585902    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c\": container with ID starting with b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c not found: ID does not exist" containerID="b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c"
Nov 01 14:42:25.585960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.585926    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c"} err="failed to get container status \"b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c\": rpc error: code = NotFound desc = could not find container \"b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c\": container with ID starting with b237e5c1c6fe42a487286c5b515dd4c2367a5628c38ad276b9cd05f633a9b23c not found: ID does not exist"
Nov 01 14:42:25.729857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.729826    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"]
Nov 01 14:42:25.781669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.781640    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-lk8d4"]
Nov 01 14:42:25.961447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.961362    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" path="/var/lib/kubelet/pods/4e4c9db2-de3a-4021-8941-a2245eced8de/volumes"
Nov 01 14:42:25.961886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:25.961872    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" path="/var/lib/kubelet/pods/4fcb770b-5f04-44f0-9cae-d882ddd526a8/volumes"
Nov 01 14:42:27.246689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:27.246652    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"]
Nov 01 14:42:27.343847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:27.343809    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"]
Nov 01 14:42:27.489523 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:27.489355    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:42:27.489523 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:27.489460    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:42:32.247838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.247774    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59" podUID="82907e08-246e-4dee-9b81-d99c7be37843" containerName="payload-processing" containerID="cri-o://2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b" gracePeriod=25
Nov 01 14:42:32.344860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.344793    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z" podUID="853493fd-adf5-4688-be1d-d267ce5523e5" containerName="payload-pre-processing" containerID="cri-o://13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a" gracePeriod=25
Nov 01 14:42:32.572659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.572632    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:42:32.692629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.692603    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:42:32.733460 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.733429    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853493fd-adf5-4688-be1d-d267ce5523e5-kube-api-access-sjb2l\" (UniqueName: \"kubernetes.io/projected/853493fd-adf5-4688-be1d-d267ce5523e5-kube-api-access-sjb2l\") pod \"853493fd-adf5-4688-be1d-d267ce5523e5\" (UID: \"853493fd-adf5-4688-be1d-d267ce5523e5\") "
Nov 01 14:42:32.733660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.733535    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853493fd-adf5-4688-be1d-d267ce5523e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853493fd-adf5-4688-be1d-d267ce5523e5-plugins-config-volume\") pod \"853493fd-adf5-4688-be1d-d267ce5523e5\" (UID: \"853493fd-adf5-4688-be1d-d267ce5523e5\") "
Nov 01 14:42:32.733660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.733567    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82907e08-246e-4dee-9b81-d99c7be37843-kube-api-access-gj478\" (UniqueName: \"kubernetes.io/projected/82907e08-246e-4dee-9b81-d99c7be37843-kube-api-access-gj478\") pod \"82907e08-246e-4dee-9b81-d99c7be37843\" (UID: \"82907e08-246e-4dee-9b81-d99c7be37843\") "
Nov 01 14:42:32.733999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.733968    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853493fd-adf5-4688-be1d-d267ce5523e5-plugins-config-volume" pod "853493fd-adf5-4688-be1d-d267ce5523e5" (UID: "853493fd-adf5-4688-be1d-d267ce5523e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:32.734165 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.733598    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82907e08-246e-4dee-9b81-d99c7be37843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82907e08-246e-4dee-9b81-d99c7be37843-plugins-config-volume\") pod \"82907e08-246e-4dee-9b81-d99c7be37843\" (UID: \"82907e08-246e-4dee-9b81-d99c7be37843\") "
Nov 01 14:42:32.734268 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.734246    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82907e08-246e-4dee-9b81-d99c7be37843-plugins-config-volume" pod "82907e08-246e-4dee-9b81-d99c7be37843" (UID: "82907e08-246e-4dee-9b81-d99c7be37843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:32.735070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.735044    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82907e08-246e-4dee-9b81-d99c7be37843-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:32.735207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.735191    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853493fd-adf5-4688-be1d-d267ce5523e5-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:32.736258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.736230    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82907e08-246e-4dee-9b81-d99c7be37843-kube-api-access-gj478" pod "82907e08-246e-4dee-9b81-d99c7be37843" (UID: "82907e08-246e-4dee-9b81-d99c7be37843"). InnerVolumeSpecName "kube-api-access-gj478". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:32.737332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.737292    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853493fd-adf5-4688-be1d-d267ce5523e5-kube-api-access-sjb2l" pod "853493fd-adf5-4688-be1d-d267ce5523e5" (UID: "853493fd-adf5-4688-be1d-d267ce5523e5"). InnerVolumeSpecName "kube-api-access-sjb2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:32.744601 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.744573    5423 generic.go:356] "Generic (PLEG): container finished" podID="853493fd-adf5-4688-be1d-d267ce5523e5" containerID="13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a" exitCode=0
Nov 01 14:42:32.744707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.744644    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"
Nov 01 14:42:32.744772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.744718    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z" event={"ID":"853493fd-adf5-4688-be1d-d267ce5523e5","Type":"ContainerDied","Data":"13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a"}
Nov 01 14:42:32.744772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.744754    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z" event={"ID":"853493fd-adf5-4688-be1d-d267ce5523e5","Type":"ContainerDied","Data":"c4bd9672502c26522d1bcc42edce16b8c7ba1a56bffcd147b3eead97cb7bc2a7"}
Nov 01 14:42:32.744880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.744775    5423 scope.go:122] "RemoveContainer" containerID="13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a"
Nov 01 14:42:32.746746 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.746723    5423 generic.go:356] "Generic (PLEG): container finished" podID="82907e08-246e-4dee-9b81-d99c7be37843" containerID="2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b" exitCode=0
Nov 01 14:42:32.746824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.746763    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59" event={"ID":"82907e08-246e-4dee-9b81-d99c7be37843","Type":"ContainerDied","Data":"2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b"}
Nov 01 14:42:32.746824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.746794    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59" event={"ID":"82907e08-246e-4dee-9b81-d99c7be37843","Type":"ContainerDied","Data":"74b009d17b41609cb9e8decc39b7e26a2214c1291f08768ec552ba5ab20735e2"}
Nov 01 14:42:32.746824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.746764    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"
Nov 01 14:42:32.755924 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.755900    5423 scope.go:122] "RemoveContainer" containerID="13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a"
Nov 01 14:42:32.756198 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:32.756178    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a\": container with ID starting with 13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a not found: ID does not exist" containerID="13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a"
Nov 01 14:42:32.756278 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.756211    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a"} err="failed to get container status \"13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a\": rpc error: code = NotFound desc = could not find container \"13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a\": container with ID starting with 13520d8b4ee39c91bf72cb7c54947d630845d3a9a80ee5c13e63404639fee80a not found: ID does not exist"
Nov 01 14:42:32.756278 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.756234    5423 scope.go:122] "RemoveContainer" containerID="2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b"
Nov 01 14:42:32.767201 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.767187    5423 scope.go:122] "RemoveContainer" containerID="2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b"
Nov 01 14:42:32.767480 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:32.767464    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b\": container with ID starting with 2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b not found: ID does not exist" containerID="2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b"
Nov 01 14:42:32.767531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.767488    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b"} err="failed to get container status \"2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b\": rpc error: code = NotFound desc = could not find container \"2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b\": container with ID starting with 2221719c90c7b3ddf4a806758f7b0066cb87bf3d6f5a5b5049ad511d3f25181b not found: ID does not exist"
Nov 01 14:42:32.836577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.836545    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjb2l\" (UniqueName: \"kubernetes.io/projected/853493fd-adf5-4688-be1d-d267ce5523e5-kube-api-access-sjb2l\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:32.836577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.836577    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj478\" (UniqueName: \"kubernetes.io/projected/82907e08-246e-4dee-9b81-d99c7be37843-kube-api-access-gj478\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:32.978859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.978827    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"]
Nov 01 14:42:32.981616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:32.981592    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a32dd3-6bc6c8948c-5dq59"]
Nov 01 14:42:33.090562 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:33.090469    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"]
Nov 01 14:42:33.129410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:33.128556    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a32dd3-56894556c5-wv98z"]
Nov 01 14:42:33.962190 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:33.962150    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82907e08-246e-4dee-9b81-d99c7be37843" path="/var/lib/kubelet/pods/82907e08-246e-4dee-9b81-d99c7be37843/volumes"
Nov 01 14:42:33.962799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:33.962778    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853493fd-adf5-4688-be1d-d267ce5523e5" path="/var/lib/kubelet/pods/853493fd-adf5-4688-be1d-d267ce5523e5/volumes"
Nov 01 14:42:34.273474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.273373    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"]
Nov 01 14:42:34.274104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274087    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy"
Nov 01 14:42:34.274147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274106    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy"
Nov 01 14:42:34.274147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274127    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy"
Nov 01 14:42:34.274147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274135    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy"
Nov 01 14:42:34.274147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274144    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy"
Nov 01 14:42:34.274274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274149    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy"
Nov 01 14:42:34.274274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274170    5423 cpu_manager.go:414] "containerMap: removing container" podUID="853493fd-adf5-4688-be1d-d267ce5523e5" containerName="payload-pre-processing"
Nov 01 14:42:34.274274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274175    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853493fd-adf5-4688-be1d-d267ce5523e5" containerName="payload-pre-processing"
Nov 01 14:42:34.274274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274205    5423 cpu_manager.go:414] "containerMap: removing container" podUID="82907e08-246e-4dee-9b81-d99c7be37843" containerName="payload-processing"
Nov 01 14:42:34.274274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274212    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82907e08-246e-4dee-9b81-d99c7be37843" containerName="payload-processing"
Nov 01 14:42:34.274442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274320    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy"
Nov 01 14:42:34.274442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274328    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcb770b-5f04-44f0-9cae-d882ddd526a8" containerName="istio-proxy"
Nov 01 14:42:34.274442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274338    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="853493fd-adf5-4688-be1d-d267ce5523e5" containerName="payload-pre-processing"
Nov 01 14:42:34.274442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274349    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="82907e08-246e-4dee-9b81-d99c7be37843" containerName="payload-processing"
Nov 01 14:42:34.274442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.274359    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4c9db2-de3a-4021-8941-a2245eced8de" containerName="istio-proxy"
Nov 01 14:42:34.279186 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.279169    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.334526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.333813    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-26a0c1-openshift-default-dockercfg-xrpzs\""
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.364491    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.364669    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8n2\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-kube-api-access-cj8n2\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.364827    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.364857    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.364901    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab4e7d0-7531-4855-b771-f36569ff5400-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.364934    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.364991    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.365036    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.372420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.365087    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.676412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.658796    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"]
Nov 01 14:42:34.679956 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.679917    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8n2\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-kube-api-access-cj8n2\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.680232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.680214    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.680538 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.680519    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.680711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.680695    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab4e7d0-7531-4855-b771-f36569ff5400-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.680842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.680828    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.681004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.680986    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.681348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.681328    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.681656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.681636    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.681845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.681831    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.710408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.684717    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.710646 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.693055    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab4e7d0-7531-4855-b771-f36569ff5400-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.710755 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.701730    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.710838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.702123    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.710922 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.702617    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.926408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.898791    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8n2\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-kube-api-access-cj8n2\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.945466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.939325    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.951842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.951809    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:34.955888 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:34.955863    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:35.151052 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:42:35.151016    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:42:35.192993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.192969    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:35.642243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.642209    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"]
Nov 01 14:42:35.679007 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.678967    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:42:35.679118 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.679051    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:42:35.679118 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.679096    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:42:35.690413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.690362    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"]
Nov 01 14:42:35.785423 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.782962    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"]
Nov 01 14:42:35.972748 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.972654    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" event={"ID":"7ab4e7d0-7531-4855-b771-f36569ff5400","Type":"ContainerStarted","Data":"2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9"}
Nov 01 14:42:35.972748 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:35.972718    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" event={"ID":"7ab4e7d0-7531-4855-b771-f36569ff5400","Type":"ContainerStarted","Data":"f4187a81e9fa1e6282eb37948d87431c11971ed3d8e62083de8d34a9debd3739"}
Nov 01 14:42:36.113407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:36.109854    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podStartSLOduration=2.109831785 podStartE2EDuration="2.109831785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:42:36.084095 +0000 UTC m=+1496.249580299" watchObservedRunningTime="2026-11-01 14:42:36.109831785 +0000 UTC m=+1496.275317111"
Nov 01 14:42:36.198827 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:36.198054    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:36.201926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:36.201885    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 14:42:36.202056 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:36.201953    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 14:42:37.226411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:37.223750    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:37.970781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:37.970749    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:37.971792 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:37.971762    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:42:40.366943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.366901    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:42:40.367436 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.366910    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:42:40.690854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.690793    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb" podUID="5927d613-42a6-4a03-a2dd-ad33d10bbd54" containerName="payload-processing" containerID="cri-o://6f9e148ea1bcadf6fe6f0b27963db9da54d822fa5ed0895e62d1cafdaf5d76b3" gracePeriod=25
Nov 01 14:42:40.784294 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.784245    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q" podUID="9b2023ad-75c9-4d6b-9b14-9186166a857a" containerName="payload-pre-processing" containerID="cri-o://b161c4cd51c87bad8149b1d7cb36e237732a83463df7335c48c332ff381461b1" gracePeriod=25
Nov 01 14:42:40.995345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.995312    5423 generic.go:356] "Generic (PLEG): container finished" podID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerID="12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea" exitCode=0
Nov 01 14:42:40.995540 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.995498    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" event={"ID":"0bb43700-7fa7-457a-be62-bb3be96e9b10","Type":"ContainerDied","Data":"12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea"}
Nov 01 14:42:40.996315 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.996018    5423 scope.go:122] "RemoveContainer" containerID="12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea"
Nov 01 14:42:40.997940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.997908    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:42:40.998049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.997991    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:42:40.998049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.998037    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:42:40.998226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.998131    5423 generic.go:356] "Generic (PLEG): container finished" podID="5927d613-42a6-4a03-a2dd-ad33d10bbd54" containerID="6f9e148ea1bcadf6fe6f0b27963db9da54d822fa5ed0895e62d1cafdaf5d76b3" exitCode=0
Nov 01 14:42:40.998288 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:40.998224    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb" event={"ID":"5927d613-42a6-4a03-a2dd-ad33d10bbd54","Type":"ContainerDied","Data":"6f9e148ea1bcadf6fe6f0b27963db9da54d822fa5ed0895e62d1cafdaf5d76b3"}
Nov 01 14:42:41.001576 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.001476    5423 generic.go:356] "Generic (PLEG): container finished" podID="9b2023ad-75c9-4d6b-9b14-9186166a857a" containerID="b161c4cd51c87bad8149b1d7cb36e237732a83463df7335c48c332ff381461b1" exitCode=0
Nov 01 14:42:41.001576 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.001505    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q" event={"ID":"9b2023ad-75c9-4d6b-9b14-9186166a857a","Type":"ContainerDied","Data":"b161c4cd51c87bad8149b1d7cb36e237732a83463df7335c48c332ff381461b1"}
Nov 01 14:42:41.081857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.081827    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:42:41.101727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.101703    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:42:41.206589 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.206518    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2023ad-75c9-4d6b-9b14-9186166a857a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2023ad-75c9-4d6b-9b14-9186166a857a-plugins-config-volume\") pod \"9b2023ad-75c9-4d6b-9b14-9186166a857a\" (UID: \"9b2023ad-75c9-4d6b-9b14-9186166a857a\") "
Nov 01 14:42:41.207012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.206977    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2023ad-75c9-4d6b-9b14-9186166a857a-kube-api-access-p27xw\" (UniqueName: \"kubernetes.io/projected/9b2023ad-75c9-4d6b-9b14-9186166a857a-kube-api-access-p27xw\") pod \"9b2023ad-75c9-4d6b-9b14-9186166a857a\" (UID: \"9b2023ad-75c9-4d6b-9b14-9186166a857a\") "
Nov 01 14:42:41.207117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.206980    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2023ad-75c9-4d6b-9b14-9186166a857a-plugins-config-volume" pod "9b2023ad-75c9-4d6b-9b14-9186166a857a" (UID: "9b2023ad-75c9-4d6b-9b14-9186166a857a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:41.207176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.207098    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5927d613-42a6-4a03-a2dd-ad33d10bbd54-kube-api-access-h79cx\" (UniqueName: \"kubernetes.io/projected/5927d613-42a6-4a03-a2dd-ad33d10bbd54-kube-api-access-h79cx\") pod \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\" (UID: \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\") "
Nov 01 14:42:41.207242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.207223    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5927d613-42a6-4a03-a2dd-ad33d10bbd54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5927d613-42a6-4a03-a2dd-ad33d10bbd54-plugins-config-volume\") pod \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\" (UID: \"5927d613-42a6-4a03-a2dd-ad33d10bbd54\") "
Nov 01 14:42:41.207721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.207680    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5927d613-42a6-4a03-a2dd-ad33d10bbd54-plugins-config-volume" pod "5927d613-42a6-4a03-a2dd-ad33d10bbd54" (UID: "5927d613-42a6-4a03-a2dd-ad33d10bbd54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:41.207976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.207927    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5927d613-42a6-4a03-a2dd-ad33d10bbd54-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:41.207976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.207953    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2023ad-75c9-4d6b-9b14-9186166a857a-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:41.209703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.209522    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2023ad-75c9-4d6b-9b14-9186166a857a-kube-api-access-p27xw" pod "9b2023ad-75c9-4d6b-9b14-9186166a857a" (UID: "9b2023ad-75c9-4d6b-9b14-9186166a857a"). InnerVolumeSpecName "kube-api-access-p27xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:41.209703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.209544    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5927d613-42a6-4a03-a2dd-ad33d10bbd54-kube-api-access-h79cx" pod "5927d613-42a6-4a03-a2dd-ad33d10bbd54" (UID: "5927d613-42a6-4a03-a2dd-ad33d10bbd54"). InnerVolumeSpecName "kube-api-access-h79cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:41.324415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.318967    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p27xw\" (UniqueName: \"kubernetes.io/projected/9b2023ad-75c9-4d6b-9b14-9186166a857a-kube-api-access-p27xw\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:41.324415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:41.319009    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79cx\" (UniqueName: \"kubernetes.io/projected/5927d613-42a6-4a03-a2dd-ad33d10bbd54-kube-api-access-h79cx\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:42.027964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.027919    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" event={"ID":"0bb43700-7fa7-457a-be62-bb3be96e9b10","Type":"ContainerStarted","Data":"fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876"}
Nov 01 14:42:42.029282 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.029261    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"
Nov 01 14:42:42.029419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.029283    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb" event={"ID":"5927d613-42a6-4a03-a2dd-ad33d10bbd54","Type":"ContainerDied","Data":"dfb9224ed8291ec7f2eee815f4726cd010230a3eeea72b20733b218a136398ef"}
Nov 01 14:42:42.029419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.029337    5423 scope.go:122] "RemoveContainer" containerID="6f9e148ea1bcadf6fe6f0b27963db9da54d822fa5ed0895e62d1cafdaf5d76b3"
Nov 01 14:42:42.030676 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.030643    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"
Nov 01 14:42:42.030783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.030686    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q" event={"ID":"9b2023ad-75c9-4d6b-9b14-9186166a857a","Type":"ContainerDied","Data":"67dcba62aedddb37ccb5f51d2fcbbc8c33663e4fb6f99a3e503eb62a06a6285c"}
Nov 01 14:42:42.039306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.039288    5423 scope.go:122] "RemoveContainer" containerID="b161c4cd51c87bad8149b1d7cb36e237732a83463df7335c48c332ff381461b1"
Nov 01 14:42:42.469440 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.431605    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"]
Nov 01 14:42:42.474624 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.472910    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466a6e-75b4c9fb9f-4ks4q"]
Nov 01 14:42:42.576408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.570628    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"]
Nov 01 14:42:42.576408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.571615    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9b2023ad-75c9-4d6b-9b14-9186166a857a" containerName="payload-pre-processing"
Nov 01 14:42:42.576408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.571639    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2023ad-75c9-4d6b-9b14-9186166a857a" containerName="payload-pre-processing"
Nov 01 14:42:42.576408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.571700    5423 cpu_manager.go:414] "containerMap: removing container" podUID="5927d613-42a6-4a03-a2dd-ad33d10bbd54" containerName="payload-processing"
Nov 01 14:42:42.576408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.571711    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5927d613-42a6-4a03-a2dd-ad33d10bbd54" containerName="payload-processing"
Nov 01 14:42:42.576408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.571877    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2023ad-75c9-4d6b-9b14-9186166a857a" containerName="payload-pre-processing"
Nov 01 14:42:42.576408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.571910    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="5927d613-42a6-4a03-a2dd-ad33d10bbd54" containerName="payload-processing"
Nov 01 14:42:42.615480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.610147    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:42.678750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.677712    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-26a0c1\""
Nov 01 14:42:42.678750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.677913    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-26a0c1-dockercfg-n4759\""
Nov 01 14:42:42.764698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.726823    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zrl\" (UniqueName: \"kubernetes.io/projected/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-kube-api-access-z5zrl\") pod \"payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5\" (UID: \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:42.764698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.727200    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5\" (UID: \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:42.765036 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.765004    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"]
Nov 01 14:42:42.838503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.830358    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zrl\" (UniqueName: \"kubernetes.io/projected/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-kube-api-access-z5zrl\") pod \"payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5\" (UID: \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:42.844416 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.838954    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5\" (UID: \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:42.887410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.846186    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5\" (UID: \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:42.906410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.903001    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"]
Nov 01 14:42:42.910405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.907304    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466a6e-85dc9fb4b7-xfhsb"]
Nov 01 14:42:42.919786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.919754    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zrl\" (UniqueName: \"kubernetes.io/projected/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-kube-api-access-z5zrl\") pod \"payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5\" (UID: \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:42.941564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.925439    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"]
Nov 01 14:42:42.947721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.947692    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:42.949261 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:42.949227    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"]
Nov 01 14:42:43.019644 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.019553    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:42:43.020484 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.020446    5423 patch_prober.go:29] interesting pod/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 01 14:42:43.020612 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.020513    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 01 14:42:43.048874 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.048842    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xn4q\" (UniqueName: \"kubernetes.io/projected/51384a52-f8b5-414b-b184-60992afb1977-kube-api-access-5xn4q\") pod \"payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb\" (UID: \"51384a52-f8b5-414b-b184-60992afb1977\") " pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:43.049281 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.049012    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51384a52-f8b5-414b-b184-60992afb1977-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb\" (UID: \"51384a52-f8b5-414b-b184-60992afb1977\") " pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:43.150873 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.150827    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xn4q\" (UniqueName: \"kubernetes.io/projected/51384a52-f8b5-414b-b184-60992afb1977-kube-api-access-5xn4q\") pod \"payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb\" (UID: \"51384a52-f8b5-414b-b184-60992afb1977\") " pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:43.151072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.150938    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51384a52-f8b5-414b-b184-60992afb1977-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb\" (UID: \"51384a52-f8b5-414b-b184-60992afb1977\") " pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:43.151701 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.151680    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51384a52-f8b5-414b-b184-60992afb1977-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb\" (UID: \"51384a52-f8b5-414b-b184-60992afb1977\") " pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:43.166428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.166381    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:43.223446 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.222656    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xn4q\" (UniqueName: \"kubernetes.io/projected/51384a52-f8b5-414b-b184-60992afb1977-kube-api-access-5xn4q\") pod \"payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb\" (UID: \"51384a52-f8b5-414b-b184-60992afb1977\") " pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:43.261318 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.261264    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:43.560658 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:42:43.560628    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6aff9c_21fa_4506_8187_e172cdd0b6a3.slice/crio-f12de89a3d797e6ef34e5f5e40237ef9f568a38d9d467b55be4a847aa0420c3e WatchSource:0}: Error finding container f12de89a3d797e6ef34e5f5e40237ef9f568a38d9d467b55be4a847aa0420c3e: Status 404 returned error can't find the container with id f12de89a3d797e6ef34e5f5e40237ef9f568a38d9d467b55be4a847aa0420c3e
Nov 01 14:42:43.563893 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.563870    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"]
Nov 01 14:42:43.660987 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:42:43.660959    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51384a52_f8b5_414b_b184_60992afb1977.slice/crio-86e577ac6da09d2eb45e7bfdb4918cd8e9d355e63e49cd902feb7c6d0063db8e WatchSource:0}: Error finding container 86e577ac6da09d2eb45e7bfdb4918cd8e9d355e63e49cd902feb7c6d0063db8e: Status 404 returned error can't find the container with id 86e577ac6da09d2eb45e7bfdb4918cd8e9d355e63e49cd902feb7c6d0063db8e
Nov 01 14:42:43.664487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.664462    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"]
Nov 01 14:42:43.961218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.961136    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5927d613-42a6-4a03-a2dd-ad33d10bbd54" path="/var/lib/kubelet/pods/5927d613-42a6-4a03-a2dd-ad33d10bbd54/volumes"
Nov 01 14:42:43.961747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:43.961725    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2023ad-75c9-4d6b-9b14-9186166a857a" path="/var/lib/kubelet/pods/9b2023ad-75c9-4d6b-9b14-9186166a857a/volumes"
Nov 01 14:42:44.020049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.019732    5423 patch_prober.go:29] interesting pod/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 01 14:42:44.020049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.019781    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 01 14:42:44.046189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.046145    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb" event={"ID":"51384a52-f8b5-414b-b184-60992afb1977","Type":"ContainerStarted","Data":"90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133"}
Nov 01 14:42:44.046368 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.046208    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb" event={"ID":"51384a52-f8b5-414b-b184-60992afb1977","Type":"ContainerStarted","Data":"86e577ac6da09d2eb45e7bfdb4918cd8e9d355e63e49cd902feb7c6d0063db8e"}
Nov 01 14:42:44.046368 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.046236    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:44.047828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.047800    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5" event={"ID":"4d6aff9c-21fa-4506-8187-e172cdd0b6a3","Type":"ContainerStarted","Data":"a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74"}
Nov 01 14:42:44.047963 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.047861    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:44.047963 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.047885    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5" event={"ID":"4d6aff9c-21fa-4506-8187-e172cdd0b6a3","Type":"ContainerStarted","Data":"f12de89a3d797e6ef34e5f5e40237ef9f568a38d9d467b55be4a847aa0420c3e"}
Nov 01 14:42:44.158988 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.158956    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:42:44.171531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.171488    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb" podStartSLOduration=2.171474573 podStartE2EDuration="2.171474573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:42:44.170490581 +0000 UTC m=+1504.335975879" watchObservedRunningTime="2026-11-01 14:42:44.171474573 +0000 UTC m=+1504.336959874"
Nov 01 14:42:44.297462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:44.294616    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5" podStartSLOduration=2.294596221 podStartE2EDuration="2.294596221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:42:44.293156519 +0000 UTC m=+1504.458641847" watchObservedRunningTime="2026-11-01 14:42:44.294596221 +0000 UTC m=+1504.460081520"
Nov 01 14:42:45.019781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:45.019740    5423 patch_prober.go:29] interesting pod/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 01 14:42:45.019991 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:45.019807    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 01 14:42:46.020041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:46.019794    5423 patch_prober.go:29] interesting pod/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 01 14:42:46.020041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:46.019868    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 01 14:42:47.013850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:47.013811    5423 patch_prober.go:29] interesting pod/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 01 14:42:47.014123 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:47.014094    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 01 14:42:48.014967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:48.014936    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:42:48.016035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:48.016016    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:42:55.080551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:55.079596    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:42:55.086332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:55.082628    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:42:55.149419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:55.148650    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:42:56.047410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.045292    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"]
Nov 01 14:42:56.085704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.085670    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.088242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.088210    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"]
Nov 01 14:42:56.169727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.169695    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-18a879-serving-cert\""
Nov 01 14:42:56.193353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.172018    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e392998c-8d13-4a07-9020-c9bf3292a2c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.193353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.191505    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.193353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.191562    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96nwk\" (UniqueName: \"kubernetes.io/projected/e392998c-8d13-4a07-9020-c9bf3292a2c4-kube-api-access-96nwk\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.193353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.191841    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-metrics-certs\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.293726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.293689    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e392998c-8d13-4a07-9020-c9bf3292a2c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.293918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.293740    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.293918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.293785    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96nwk\" (UniqueName: \"kubernetes.io/projected/e392998c-8d13-4a07-9020-c9bf3292a2c4-kube-api-access-96nwk\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.294049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.294027    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-metrics-certs\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.294537 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.294511    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e392998c-8d13-4a07-9020-c9bf3292a2c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.296296 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.296255    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.296433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.296348    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-metrics-certs\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.354645 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.354616    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96nwk\" (UniqueName: \"kubernetes.io/projected/e392998c-8d13-4a07-9020-c9bf3292a2c4-kube-api-access-96nwk\") pod \"maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:56.435475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:56.435440    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:57.044441 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:42:57.044408    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode392998c_8d13_4a07_9020_c9bf3292a2c4.slice/crio-a46716ff3c1604ebf276658ce58f083327ed1fb316ce8f596194662dd2e96dae WatchSource:0}: Error finding container a46716ff3c1604ebf276658ce58f083327ed1fb316ce8f596194662dd2e96dae: Status 404 returned error can't find the container with id a46716ff3c1604ebf276658ce58f083327ed1fb316ce8f596194662dd2e96dae
Nov 01 14:42:57.047548 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:57.047528    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"]
Nov 01 14:42:57.185535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:57.185499    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4" event={"ID":"e392998c-8d13-4a07-9020-c9bf3292a2c4","Type":"ContainerStarted","Data":"a46716ff3c1604ebf276658ce58f083327ed1fb316ce8f596194662dd2e96dae"}
Nov 01 14:42:58.194431 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:58.194305    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4" event={"ID":"e392998c-8d13-4a07-9020-c9bf3292a2c4","Type":"ContainerStarted","Data":"51e8e408c654096e1d6ab74216a74f2a7e0115abc0eade4f515cfa72d750f94f"}
Nov 01 14:42:58.194829 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:58.194430    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:42:58.315830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:42:58.315773    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4" podStartSLOduration=2.418402569 podStartE2EDuration="3.315758877s" totalImagesPullingTime="897.356308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:42:58.312721422 +0000 UTC m=+1518.478206723" watchObservedRunningTime="2026-11-01 14:42:58.315758877 +0000 UTC m=+1518.481244174"
Nov 01 14:43:04.206882 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:04.206853    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:43:05.149227 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:43:05.149194    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:43:08.387491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:08.387462    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"]
Nov 01 14:43:13.388655 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:13.388591    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg" podUID="bc292128-7480-4da8-b594-2f65538a5e5a" containerName="payload-pre-processing" containerID="cri-o://2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c" gracePeriod=25
Nov 01 14:43:13.661562 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:13.661533    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:43:13.771450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:13.771412    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc292128-7480-4da8-b594-2f65538a5e5a-kube-api-access-j22rw\" (UniqueName: \"kubernetes.io/projected/bc292128-7480-4da8-b594-2f65538a5e5a-kube-api-access-j22rw\") pod \"bc292128-7480-4da8-b594-2f65538a5e5a\" (UID: \"bc292128-7480-4da8-b594-2f65538a5e5a\") "
Nov 01 14:43:13.771633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:13.771481    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc292128-7480-4da8-b594-2f65538a5e5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc292128-7480-4da8-b594-2f65538a5e5a-plugins-config-volume\") pod \"bc292128-7480-4da8-b594-2f65538a5e5a\" (UID: \"bc292128-7480-4da8-b594-2f65538a5e5a\") "
Nov 01 14:43:13.771865 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:13.771843    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc292128-7480-4da8-b594-2f65538a5e5a-plugins-config-volume" pod "bc292128-7480-4da8-b594-2f65538a5e5a" (UID: "bc292128-7480-4da8-b594-2f65538a5e5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:43:13.772173 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:13.772152    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc292128-7480-4da8-b594-2f65538a5e5a-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:13.773648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:13.773618    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc292128-7480-4da8-b594-2f65538a5e5a-kube-api-access-j22rw" pod "bc292128-7480-4da8-b594-2f65538a5e5a" (UID: "bc292128-7480-4da8-b594-2f65538a5e5a"). InnerVolumeSpecName "kube-api-access-j22rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:13.873105 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:13.873069    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j22rw\" (UniqueName: \"kubernetes.io/projected/bc292128-7480-4da8-b594-2f65538a5e5a-kube-api-access-j22rw\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:14.296805 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.296716    5423 generic.go:356] "Generic (PLEG): container finished" podID="bc292128-7480-4da8-b594-2f65538a5e5a" containerID="2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c" exitCode=0
Nov 01 14:43:14.296805 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.296754    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg" event={"ID":"bc292128-7480-4da8-b594-2f65538a5e5a","Type":"ContainerDied","Data":"2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c"}
Nov 01 14:43:14.296805 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.296773    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"
Nov 01 14:43:14.297055 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.296811    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg" event={"ID":"bc292128-7480-4da8-b594-2f65538a5e5a","Type":"ContainerDied","Data":"bf41cd4c3b20fd861e13a2dc23b2298cf0ca6bf228b46ed0c20a4c791998ce3d"}
Nov 01 14:43:14.297055 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.296838    5423 scope.go:122] "RemoveContainer" containerID="2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c"
Nov 01 14:43:14.306606 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.306583    5423 scope.go:122] "RemoveContainer" containerID="2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c"
Nov 01 14:43:14.306839 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:43:14.306820    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c\": container with ID starting with 2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c not found: ID does not exist" containerID="2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c"
Nov 01 14:43:14.306906 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.306850    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c"} err="failed to get container status \"2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c\": rpc error: code = NotFound desc = could not find container \"2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c\": container with ID starting with 2b7ae3da82443f0fb3e95c40279706875cd4befd4d022d2d09ffbf0ed7c5370c not found: ID does not exist"
Nov 01 14:43:14.470121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.470083    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"]
Nov 01 14:43:14.519246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:14.519217    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-116e85-6fb46f9cc9-vt2lg"]
Nov 01 14:43:16.162960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:16.151149    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc292128-7480-4da8-b594-2f65538a5e5a" path="/var/lib/kubelet/pods/bc292128-7480-4da8-b594-2f65538a5e5a/volumes"
Nov 01 14:43:23.259860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:23.259790    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"]
Nov 01 14:43:23.260254 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:23.259985    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy" containerID="cri-o://e925c272c485cdf70171ec58c30cbee70a884c3360c8c5436117aa4c045ee39a" gracePeriod=30
Nov 01 14:43:28.471003 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.470969    5423 generic.go:356] "Generic (PLEG): container finished" podID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerID="e925c272c485cdf70171ec58c30cbee70a884c3360c8c5436117aa4c045ee39a" exitCode=0
Nov 01 14:43:28.471526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.471174    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" event={"ID":"da8aec01-28a3-4dba-81b8-c63eb0a32011","Type":"ContainerDied","Data":"e925c272c485cdf70171ec58c30cbee70a884c3360c8c5436117aa4c045ee39a"}
Nov 01 14:43:28.471526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.471234    5423 scope.go:122] "RemoveContainer" containerID="13ce3ccdc16bee8e06940526b633636bd9a7ded3483f1a349acf354f7be8f8df"
Nov 01 14:43:28.531427 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.531358    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.687773    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-envoy\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.687833    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-data\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.687895    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-kube-api-access-hktpp\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-kube-api-access-hktpp\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.688047    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-socket\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.688112    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8aec01-28a3-4dba-81b8-c63eb0a32011-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8aec01-28a3-4dba-81b8-c63eb0a32011-istiod-ca-cert\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.688145    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-podinfo\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.688285    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-credential-socket\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.688487    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-token\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-token\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.689030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.688549    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-certs\") pod \"da8aec01-28a3-4dba-81b8-c63eb0a32011\" (UID: \"da8aec01-28a3-4dba-81b8-c63eb0a32011\") "
Nov 01 14:43:28.696517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.696148    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-certs" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:28.714419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.707356    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-data" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:28.722900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.716260    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-envoy" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:28.722900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.721687    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-podinfo" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:43:28.722900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.721972    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-socket" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:28.722900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.722218    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-credential-socket" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:28.723373 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.723337    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8aec01-28a3-4dba-81b8-c63eb0a32011-istiod-ca-cert" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:43:28.744511 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.743644    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-kube-api-access-hktpp" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "kube-api-access-hktpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:28.763636 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.763594    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-token" pod "da8aec01-28a3-4dba-81b8-c63eb0a32011" (UID: "da8aec01-28a3-4dba-81b8-c63eb0a32011"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:28.802516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.802345    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:28.802967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.802787    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:28.802967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.802817    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:28.802967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.802830    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:28.802967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.802845    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:28.802967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.802863    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktpp\" (UniqueName: \"kubernetes.io/projected/da8aec01-28a3-4dba-81b8-c63eb0a32011-kube-api-access-hktpp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:28.802967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.802881    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8aec01-28a3-4dba-81b8-c63eb0a32011-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:28.803681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.803417    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8aec01-28a3-4dba-81b8-c63eb0a32011-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:28.803681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:28.803451    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8aec01-28a3-4dba-81b8-c63eb0a32011-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.478671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:29.478630    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml" event={"ID":"da8aec01-28a3-4dba-81b8-c63eb0a32011","Type":"ContainerDied","Data":"0924f216ac77fdab51d09ebe0ca6f1684dd63a18121dd1724adac7bab1cf19eb"}
Nov 01 14:43:29.478671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:29.478657    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"
Nov 01 14:43:29.479171 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:29.478692    5423 scope.go:122] "RemoveContainer" containerID="e925c272c485cdf70171ec58c30cbee70a884c3360c8c5436117aa4c045ee39a"
Nov 01 14:43:29.699299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:29.699265    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"]
Nov 01 14:43:29.749063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:29.749030    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a32dd3-openshift-default-799c9c8457-pdzml"]
Nov 01 14:43:29.965630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:29.965558    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" path="/var/lib/kubelet/pods/da8aec01-28a3-4dba-81b8-c63eb0a32011/volumes"
Nov 01 14:43:33.650464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:33.650432    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"]
Nov 01 14:43:33.650830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:33.650628    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy" containerID="cri-o://2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9" gracePeriod=30
Nov 01 14:43:35.150141 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:43:35.150080    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:43:38.913570 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.913513    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:43:38.967049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967011    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-credential-socket\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967219 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967172    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c5417f6-4949-4efd-8429-69374e520c77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5417f6-4949-4efd-8429-69374e520c77-istio-podinfo\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967285 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967229    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-kube-api-access-7tz7r\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-kube-api-access-7tz7r\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967275    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-credential-socket" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:38.967336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967293    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-istio-token\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-istio-token\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967360    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-socket\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967452    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5417f6-4949-4efd-8429-69374e520c77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5417f6-4949-4efd-8429-69374e520c77-istiod-ca-cert\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967589 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967521    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-data\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967589 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967561    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-envoy\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967607    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-certs\") pod \"2c5417f6-4949-4efd-8429-69374e520c77\" (UID: \"2c5417f6-4949-4efd-8429-69374e520c77\") "
Nov 01 14:43:38.967698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967618    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-socket" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:38.967888 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967864    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-data" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:38.967970 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967877    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-certs" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:38.967970 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.967918    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5417f6-4949-4efd-8429-69374e520c77-istiod-ca-cert" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:43:38.968293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.968270    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:38.968350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.968301    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c5417f6-4949-4efd-8429-69374e520c77-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:38.968350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.968317    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:38.968350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.968332    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:38.968350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.968346    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:38.969674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.969652    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-envoy" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:38.969765 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.969713    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-istio-token" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:38.969862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.969846    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-kube-api-access-7tz7r" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "kube-api-access-7tz7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:38.969920 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:38.969900    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c5417f6-4949-4efd-8429-69374e520c77-istio-podinfo" pod "2c5417f6-4949-4efd-8429-69374e520c77" (UID: "2c5417f6-4949-4efd-8429-69374e520c77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:43:39.069769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.069729    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.069769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.069755    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c5417f6-4949-4efd-8429-69374e520c77-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.069769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.069766    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c5417f6-4949-4efd-8429-69374e520c77-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.069992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.069777    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tz7r\" (UniqueName: \"kubernetes.io/projected/2c5417f6-4949-4efd-8429-69374e520c77-kube-api-access-7tz7r\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.660047 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.660015    5423 generic.go:356] "Generic (PLEG): container finished" podID="2c5417f6-4949-4efd-8429-69374e520c77" containerID="2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9" exitCode=0
Nov 01 14:43:39.660218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.660068    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"
Nov 01 14:43:39.660303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.660222    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" event={"ID":"2c5417f6-4949-4efd-8429-69374e520c77","Type":"ContainerDied","Data":"2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9"}
Nov 01 14:43:39.660303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.660273    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv" event={"ID":"2c5417f6-4949-4efd-8429-69374e520c77","Type":"ContainerDied","Data":"184813d5e5dc14005148a01b8ef434a29b8f901e87daf656ee97dae366c6baae"}
Nov 01 14:43:39.660530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.660305    5423 scope.go:122] "RemoveContainer" containerID="2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9"
Nov 01 14:43:39.672236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.672220    5423 scope.go:122] "RemoveContainer" containerID="2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9"
Nov 01 14:43:39.672553 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:43:39.672530    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9\": container with ID starting with 2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9 not found: ID does not exist" containerID="2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9"
Nov 01 14:43:39.672622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.672562    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9"} err="failed to get container status \"2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9\": rpc error: code = NotFound desc = could not find container \"2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9\": container with ID starting with 2045006b91e1a7270b24abf0c79a712a110cd71d7e5c1524df0a68b77dd464a9 not found: ID does not exist"
Nov 01 14:43:39.842907 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.842873    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"]
Nov 01 14:43:39.846536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.846511    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-d4tmv"]
Nov 01 14:43:39.961847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:39.961777    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5417f6-4949-4efd-8429-69374e520c77" path="/var/lib/kubelet/pods/2c5417f6-4949-4efd-8429-69374e520c77/volumes"
Nov 01 14:43:43.891129 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:43.889876    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:43:50.008056 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.008021    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"]
Nov 01 14:43:50.009669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.009640    5423 cpu_manager.go:414] "containerMap: removing container" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy"
Nov 01 14:43:50.009819 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.009804    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy"
Nov 01 14:43:50.009969 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.009955    5423 cpu_manager.go:414] "containerMap: removing container" podUID="bc292128-7480-4da8-b594-2f65538a5e5a" containerName="payload-pre-processing"
Nov 01 14:43:50.010070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.010058    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc292128-7480-4da8-b594-2f65538a5e5a" containerName="payload-pre-processing"
Nov 01 14:43:50.010186 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.010175    5423 cpu_manager.go:414] "containerMap: removing container" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy"
Nov 01 14:43:50.010269 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.010258    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy"
Nov 01 14:43:50.010353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.010342    5423 cpu_manager.go:414] "containerMap: removing container" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy"
Nov 01 14:43:50.010454 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.010442    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy"
Nov 01 14:43:50.026483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.010699    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5417f6-4949-4efd-8429-69374e520c77" containerName="istio-proxy"
Nov 01 14:43:50.026483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.025271    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc292128-7480-4da8-b594-2f65538a5e5a" containerName="payload-pre-processing"
Nov 01 14:43:50.026483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.025314    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy"
Nov 01 14:43:50.026483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.025326    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8aec01-28a3-4dba-81b8-c63eb0a32011" containerName="istio-proxy"
Nov 01 14:43:50.029531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.029512    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.076784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.076745    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.076945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.076797    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.076945 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.076912    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.077040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.076950    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jph7t\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-kube-api-access-jph7t\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.077127 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.077111    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.077190 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.077164    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.077327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.077312    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.077408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.077362    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.077523 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.077508    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.135473 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.135436    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"]
Nov 01 14:43:50.179321 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179282    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179336    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179446    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179491    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179552    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179575    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179593    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179630    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179651    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jph7t\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-kube-api-access-jph7t\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179679    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179874    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.179998 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.179975    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.180299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.180273    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.180730 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.180703    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.182277 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.182254    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.182364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.182260    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.237200 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.237164    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.237515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.237492    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jph7t\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-kube-api-access-jph7t\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.342948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.342907    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:50.777443 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:43:50.777414    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12fcf30_ae85_4ba7_8228_768491bd5ee3.slice/crio-d786d94456f264f7aac2b6925c5858a3610d74d1804e3ba72fa858254c6c1fea WatchSource:0}: Error finding container d786d94456f264f7aac2b6925c5858a3610d74d1804e3ba72fa858254c6c1fea: Status 404 returned error can't find the container with id d786d94456f264f7aac2b6925c5858a3610d74d1804e3ba72fa858254c6c1fea
Nov 01 14:43:50.780136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.780104    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:43:50.780226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.780172    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:43:50.780226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.780200    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:43:50.780500 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.780477    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"]
Nov 01 14:43:50.868911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.868876    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" event={"ID":"a12fcf30-ae85-4ba7-8228-768491bd5ee3","Type":"ContainerStarted","Data":"d786d94456f264f7aac2b6925c5858a3610d74d1804e3ba72fa858254c6c1fea"}
Nov 01 14:43:50.987364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:50.987303    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podStartSLOduration=1.9872894620000001 podStartE2EDuration="1.987289462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:50.984706514 +0000 UTC m=+1571.150191833" watchObservedRunningTime="2026-11-01 14:43:50.987289462 +0000 UTC m=+1571.152774759"
Nov 01 14:43:51.343412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:51.343358    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:51.345273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:51.345244    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 14:43:51.345429 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:51.345292    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 14:43:51.882412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:51.882074    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" event={"ID":"a12fcf30-ae85-4ba7-8228-768491bd5ee3","Type":"ContainerStarted","Data":"1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f"}
Nov 01 14:43:52.357944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:52.357899    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 14:43:52.358440 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:52.357961    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 14:43:53.343885 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:53.343849    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 14:43:53.344088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:53.343907    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 14:43:54.042408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:54.041856    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:43:54.042408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:54.041952    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:43:54.349258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:54.348741    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 14:43:54.349258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:54.348812    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 14:43:55.357340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:55.357278    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 14:43:55.357907 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:55.357347    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 14:43:56.358408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:56.343788    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 14:43:56.358408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:56.343862    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 14:43:56.963997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:56.963967    5423 generic.go:356] "Generic (PLEG): container finished" podID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerID="2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9" exitCode=0
Nov 01 14:43:56.964207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:56.964048    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" event={"ID":"7ab4e7d0-7531-4855-b771-f36569ff5400","Type":"ContainerDied","Data":"2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9"}
Nov 01 14:43:56.964488 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:56.964474    5423 scope.go:122] "RemoveContainer" containerID="2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9"
Nov 01 14:43:56.965986 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:56.965956    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:43:56.966098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:56.966038    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:43:56.966165 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:56.966105    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:43:57.265862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:57.265830    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"]
Nov 01 14:43:57.266486 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:57.266454    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy" containerID="cri-o://85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5" gracePeriod=30
Nov 01 14:43:57.343844 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:57.343805    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 14:43:57.343998 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:57.343902    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 14:43:58.032108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:58.032068    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" event={"ID":"7ab4e7d0-7531-4855-b771-f36569ff5400","Type":"ContainerStarted","Data":"f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce"}
Nov 01 14:43:58.193218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:58.193182    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:43:58.193377 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:58.193345    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 14:43:58.193468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:58.193436    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 14:43:58.348895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:58.348864    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:58.349350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:58.349331    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:58.350051 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:58.350035    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:43:59.215139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:59.215101    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 14:43:59.215723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:43:59.215686    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 14:44:00.287363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:00.286531    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 14:44:00.287363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:00.286590    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 14:44:01.206410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:01.194569    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 14:44:01.206410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:01.194648    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 14:44:01.823204 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:01.823165    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"]
Nov 01 14:44:01.919297 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:01.919258    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"]
Nov 01 14:44:02.194123 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.194039    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 14:44:02.194123 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.194098    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 14:44:02.632926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.632898    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:44:02.716411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.715799    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-kube-api-access-45bz6\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-kube-api-access-45bz6\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.716411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.715895    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-podinfo\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.716411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.715931    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a3901c-5d4f-4378-9778-e4186b9863e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a3901c-5d4f-4378-9778-e4186b9863e3-istiod-ca-cert\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.716411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.716025    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-token\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-token\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.716411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.716222    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-certs\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.716411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.716258    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-data\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.716411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.716300    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-envoy\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.716411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.716362    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-credential-socket\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.716968 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.716716    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-socket\") pod \"55a3901c-5d4f-4378-9778-e4186b9863e3\" (UID: \"55a3901c-5d4f-4378-9778-e4186b9863e3\") "
Nov 01 14:44:02.719530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.719493    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-credential-socket" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:02.719667 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.719642    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-socket" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:02.720437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.720343    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-certs" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:02.720729 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.720707    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-data" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:02.720871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.720817    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a3901c-5d4f-4378-9778-e4186b9863e3-istiod-ca-cert" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:02.723165 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.723118    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-podinfo" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:02.723260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.723189    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-token" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:02.723260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.723237    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-kube-api-access-45bz6" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "kube-api-access-45bz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:02.723347 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.723260    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-envoy" pod "55a3901c-5d4f-4378-9778-e4186b9863e3" (UID: "55a3901c-5d4f-4378-9778-e4186b9863e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:02.820747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820702    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:02.820747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820742    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:02.820747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820755    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:02.821032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820769    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:02.821032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820782    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a3901c-5d4f-4378-9778-e4186b9863e3-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:02.821032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820796    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45bz6\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-kube-api-access-45bz6\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:02.821032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820809    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:02.821032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820823    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a3901c-5d4f-4378-9778-e4186b9863e3-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:02.821032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:02.820835    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a3901c-5d4f-4378-9778-e4186b9863e3-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:03.203475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.203368    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 14:44:03.225601 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.225560    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 14:44:03.345232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.333740    5423 generic.go:356] "Generic (PLEG): container finished" podID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerID="85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5" exitCode=0
Nov 01 14:44:03.345232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.333897    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" event={"ID":"55a3901c-5d4f-4378-9778-e4186b9863e3","Type":"ContainerDied","Data":"85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5"}
Nov 01 14:44:03.345232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.333952    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw" event={"ID":"55a3901c-5d4f-4378-9778-e4186b9863e3","Type":"ContainerDied","Data":"60391d929d70d07d20089060832e59e917e8c83fa71e6c1f1c1612590de2fb44"}
Nov 01 14:44:03.345232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.333983    5423 scope.go:122] "RemoveContainer" containerID="85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5"
Nov 01 14:44:03.345232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.334151    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"
Nov 01 14:44:03.633720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.633306    5423 scope.go:122] "RemoveContainer" containerID="25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e"
Nov 01 14:44:03.741790 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.741617    5423 scope.go:122] "RemoveContainer" containerID="85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5"
Nov 01 14:44:03.748777 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:03.743222    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5\": container with ID starting with 85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5 not found: ID does not exist" containerID="85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5"
Nov 01 14:44:03.748777 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.743263    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5"} err="failed to get container status \"85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5\": rpc error: code = NotFound desc = could not find container \"85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5\": container with ID starting with 85f74e5491167048d32ad612d14199adffb6fd8815528eeefd018123f3e8d3c5 not found: ID does not exist"
Nov 01 14:44:03.748777 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.743289    5423 scope.go:122] "RemoveContainer" containerID="25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e"
Nov 01 14:44:03.754417 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:03.753530    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e\": container with ID starting with 25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e not found: ID does not exist" containerID="25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e"
Nov 01 14:44:03.754417 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.753580    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e"} err="failed to get container status \"25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e\": rpc error: code = NotFound desc = could not find container \"25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e\": container with ID starting with 25e7c260ef21e4f2f7f70b6ddd640c4b43cfe1fd87a6e08694b660df47909c3e not found: ID does not exist"
Nov 01 14:44:03.758407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.758022    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"]
Nov 01 14:44:03.818458 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.818426    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-116e85-openshift-default-6dd7f5df8c-mpjqw"]
Nov 01 14:44:03.964221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:03.964131    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" path="/var/lib/kubelet/pods/55a3901c-5d4f-4378-9778-e4186b9863e3/volumes"
Nov 01 14:44:04.235266 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:04.235185    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:44:04.235746 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:04.235660    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:44:04.237061 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:04.237038    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:44:05.225412 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:05.221680    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:44:06.823961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:06.823900    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59" podUID="e9a04f92-ec68-40e5-9a2b-39bdbf4919be" containerName="payload-processing" containerID="cri-o://f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7" gracePeriod=25
Nov 01 14:44:06.921033 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:06.920992    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r" podUID="ac7599a1-3847-46aa-874d-4a738de877e0" containerName="payload-pre-processing" containerID="cri-o://f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e" gracePeriod=25
Nov 01 14:44:07.200257 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.200230    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:44:07.236205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.236182    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:44:07.300177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.300144    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-kube-api-access-qzq47\" (UniqueName: \"kubernetes.io/projected/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-kube-api-access-qzq47\") pod \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\" (UID: \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\") "
Nov 01 14:44:07.300340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.300286    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7599a1-3847-46aa-874d-4a738de877e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7599a1-3847-46aa-874d-4a738de877e0-plugins-config-volume\") pod \"ac7599a1-3847-46aa-874d-4a738de877e0\" (UID: \"ac7599a1-3847-46aa-874d-4a738de877e0\") "
Nov 01 14:44:07.300340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.300327    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-plugins-config-volume\") pod \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\" (UID: \"e9a04f92-ec68-40e5-9a2b-39bdbf4919be\") "
Nov 01 14:44:07.300513 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.300495    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7599a1-3847-46aa-874d-4a738de877e0-kube-api-access-f2pzt\" (UniqueName: \"kubernetes.io/projected/ac7599a1-3847-46aa-874d-4a738de877e0-kube-api-access-f2pzt\") pod \"ac7599a1-3847-46aa-874d-4a738de877e0\" (UID: \"ac7599a1-3847-46aa-874d-4a738de877e0\") "
Nov 01 14:44:07.300733 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.300708    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7599a1-3847-46aa-874d-4a738de877e0-plugins-config-volume" pod "ac7599a1-3847-46aa-874d-4a738de877e0" (UID: "ac7599a1-3847-46aa-874d-4a738de877e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:07.300825 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.300741    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-plugins-config-volume" pod "e9a04f92-ec68-40e5-9a2b-39bdbf4919be" (UID: "e9a04f92-ec68-40e5-9a2b-39bdbf4919be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:07.301066 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.301050    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7599a1-3847-46aa-874d-4a738de877e0-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:07.301108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.301069    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:07.302332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.302311    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-kube-api-access-qzq47" pod "e9a04f92-ec68-40e5-9a2b-39bdbf4919be" (UID: "e9a04f92-ec68-40e5-9a2b-39bdbf4919be"). InnerVolumeSpecName "kube-api-access-qzq47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:07.302462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.302441    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7599a1-3847-46aa-874d-4a738de877e0-kube-api-access-f2pzt" pod "ac7599a1-3847-46aa-874d-4a738de877e0" (UID: "ac7599a1-3847-46aa-874d-4a738de877e0"). InnerVolumeSpecName "kube-api-access-f2pzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:07.361516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.361417    5423 generic.go:356] "Generic (PLEG): container finished" podID="e9a04f92-ec68-40e5-9a2b-39bdbf4919be" containerID="f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7" exitCode=0
Nov 01 14:44:07.361516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.361508    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"
Nov 01 14:44:07.361745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.361526    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59" event={"ID":"e9a04f92-ec68-40e5-9a2b-39bdbf4919be","Type":"ContainerDied","Data":"f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7"}
Nov 01 14:44:07.361745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.361619    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59" event={"ID":"e9a04f92-ec68-40e5-9a2b-39bdbf4919be","Type":"ContainerDied","Data":"04eff6c78eb557f73d59c745088f96031f22df81c3c4ec08a8ea4f6f2d824da4"}
Nov 01 14:44:07.361745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.361648    5423 scope.go:122] "RemoveContainer" containerID="f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7"
Nov 01 14:44:07.363184 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.363156    5423 generic.go:356] "Generic (PLEG): container finished" podID="ac7599a1-3847-46aa-874d-4a738de877e0" containerID="f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e" exitCode=0
Nov 01 14:44:07.363310 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.363215    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r" event={"ID":"ac7599a1-3847-46aa-874d-4a738de877e0","Type":"ContainerDied","Data":"f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e"}
Nov 01 14:44:07.363310 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.363254    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r" event={"ID":"ac7599a1-3847-46aa-874d-4a738de877e0","Type":"ContainerDied","Data":"d56d507b31e2fb0d007908ab1297b73f166da42308291699c8a718b30afb0b65"}
Nov 01 14:44:07.363310 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.363218    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"
Nov 01 14:44:07.372372 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.372348    5423 scope.go:122] "RemoveContainer" containerID="f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7"
Nov 01 14:44:07.372754 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:07.372723    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7\": container with ID starting with f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7 not found: ID does not exist" containerID="f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7"
Nov 01 14:44:07.372833 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.372763    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7"} err="failed to get container status \"f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7\": rpc error: code = NotFound desc = could not find container \"f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7\": container with ID starting with f6fcbb4bbed50d32abbe70b7e1df7f04fc2a6bc82a106021f028c7146ca255e7 not found: ID does not exist"
Nov 01 14:44:07.372833 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.372785    5423 scope.go:122] "RemoveContainer" containerID="f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e"
Nov 01 14:44:07.381961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.381946    5423 scope.go:122] "RemoveContainer" containerID="f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e"
Nov 01 14:44:07.382246 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:07.382222    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e\": container with ID starting with f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e not found: ID does not exist" containerID="f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e"
Nov 01 14:44:07.382429 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.382251    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e"} err="failed to get container status \"f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e\": rpc error: code = NotFound desc = could not find container \"f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e\": container with ID starting with f01f69b4da0f8adfe617f29ce8482a8075d9018abd96aff2b28ec920c8e8d74e not found: ID does not exist"
Nov 01 14:44:07.402193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.402168    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2pzt\" (UniqueName: \"kubernetes.io/projected/ac7599a1-3847-46aa-874d-4a738de877e0-kube-api-access-f2pzt\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:07.402193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.402193    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzq47\" (UniqueName: \"kubernetes.io/projected/e9a04f92-ec68-40e5-9a2b-39bdbf4919be-kube-api-access-qzq47\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:07.547398 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.547356    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"]
Nov 01 14:44:07.550094 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.550067    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-6d85db49dc-v8s2r"]
Nov 01 14:44:07.659422 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.659309    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"]
Nov 01 14:44:07.662705 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.662676    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-7ff455449c-gls59"]
Nov 01 14:44:07.962169 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.962050    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7599a1-3847-46aa-874d-4a738de877e0" path="/var/lib/kubelet/pods/ac7599a1-3847-46aa-874d-4a738de877e0/volumes"
Nov 01 14:44:07.962703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:07.962677    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a04f92-ec68-40e5-9a2b-39bdbf4919be" path="/var/lib/kubelet/pods/e9a04f92-ec68-40e5-9a2b-39bdbf4919be/volumes"
Nov 01 14:44:14.337514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:14.337478    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:44:34.157412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:34.156742    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"]
Nov 01 14:44:34.157412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:34.157040    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy" containerID="cri-o://c9259cdf042adca33c63acd0c8e6937b3b8dcf1494920d9afd61f66e65ec3350" gracePeriod=30
Nov 01 14:44:34.196132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:34.195490    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"]
Nov 01 14:44:34.196132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:34.195744    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy" containerID="cri-o://658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6" gracePeriod=30
Nov 01 14:44:34.202615 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:34.202578    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"]
Nov 01 14:44:34.202835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:34.202808    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy" containerID="cri-o://0f8fafbace9dbcef93425f99fe02b139e8db5478e9d0930cd7a5c632f113f6ec" gracePeriod=30
Nov 01 14:44:34.214060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:34.213997    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"]
Nov 01 14:44:34.214526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:34.214481    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy" containerID="cri-o://ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6" gracePeriod=30
Nov 01 14:44:35.150866 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:35.150824    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:44:38.866568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:38.866527    5423 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:44:38.866955 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:38.866596    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:44:39.509603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.509571    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:44:39.534591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.534566    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:44:39.570780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.570746    5423 generic.go:356] "Generic (PLEG): container finished" podID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerID="0f8fafbace9dbcef93425f99fe02b139e8db5478e9d0930cd7a5c632f113f6ec" exitCode=0
Nov 01 14:44:39.570950 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.570865    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" event={"ID":"904a66ff-67b2-4a86-8756-ceeb67668dff","Type":"ContainerDied","Data":"0f8fafbace9dbcef93425f99fe02b139e8db5478e9d0930cd7a5c632f113f6ec"}
Nov 01 14:44:39.573079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.573049    5423 generic.go:356] "Generic (PLEG): container finished" podID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerID="c9259cdf042adca33c63acd0c8e6937b3b8dcf1494920d9afd61f66e65ec3350" exitCode=0
Nov 01 14:44:39.573246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.573207    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" event={"ID":"82556296-1a82-4ac7-828c-b8a223aa3fd7","Type":"ContainerDied","Data":"c9259cdf042adca33c63acd0c8e6937b3b8dcf1494920d9afd61f66e65ec3350"}
Nov 01 14:44:39.575346 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.575323    5423 generic.go:356] "Generic (PLEG): container finished" podID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerID="ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6" exitCode=0
Nov 01 14:44:39.575505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.575409    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" event={"ID":"7dda68f1-989b-4643-9013-0e2962c9b4d9","Type":"ContainerDied","Data":"ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6"}
Nov 01 14:44:39.575505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.575454    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf" event={"ID":"7dda68f1-989b-4643-9013-0e2962c9b4d9","Type":"ContainerDied","Data":"3fc763e921274cea9c2b15975198e52fb5ca53022c9620b23f4d33ef3eae3506"}
Nov 01 14:44:39.575505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.575477    5423 scope.go:122] "RemoveContainer" containerID="ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6"
Nov 01 14:44:39.575505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.575483    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"
Nov 01 14:44:39.579509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.578971    5423 generic.go:356] "Generic (PLEG): container finished" podID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerID="658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6" exitCode=0
Nov 01 14:44:39.579509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.579018    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" event={"ID":"1ba6a000-311b-4f1f-97f6-298bb0e2f87e","Type":"ContainerDied","Data":"658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6"}
Nov 01 14:44:39.579509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.579058    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm" event={"ID":"1ba6a000-311b-4f1f-97f6-298bb0e2f87e","Type":"ContainerDied","Data":"d990b0652ec752b5f2583e0f255a28575ab49477477f739699f0b7c5053bfc18"}
Nov 01 14:44:39.579509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.579135    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"
Nov 01 14:44:39.588124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.588107    5423 scope.go:122] "RemoveContainer" containerID="e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20"
Nov 01 14:44:39.598669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.598646    5423 scope.go:122] "RemoveContainer" containerID="ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6"
Nov 01 14:44:39.599234 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:39.599204    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6\": container with ID starting with ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6 not found: ID does not exist" containerID="ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6"
Nov 01 14:44:39.599341 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.599244    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6"} err="failed to get container status \"ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6\": rpc error: code = NotFound desc = could not find container \"ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6\": container with ID starting with ef82413b258a25b8a4d0e2f416ac12a6b0262306ac58a5e066ffef1ed67f87e6 not found: ID does not exist"
Nov 01 14:44:39.599341 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.599267    5423 scope.go:122] "RemoveContainer" containerID="e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20"
Nov 01 14:44:39.599923 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:39.599904    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20\": container with ID starting with e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20 not found: ID does not exist" containerID="e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20"
Nov 01 14:44:39.600228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.599930    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20"} err="failed to get container status \"e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20\": rpc error: code = NotFound desc = could not find container \"e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20\": container with ID starting with e3a5a74a6d7b9b9ceac3f5d8f92ec9e934be9f6e7551969a16044a32cf9fdb20 not found: ID does not exist"
Nov 01 14:44:39.600228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.599952    5423 scope.go:122] "RemoveContainer" containerID="658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6"
Nov 01 14:44:39.606074 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606053    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-envoy\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.606174 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606096    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-credential-socket\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.606174 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606151    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-socket\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.606174 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606171    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-podinfo\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.606345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606271    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-token\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-token\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.606486 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606465    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-socket" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.606560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606488    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-credential-socket" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.606560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606498    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-kube-api-access-jb9s2\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-kube-api-access-jb9s2\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.606560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606541    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istiod-ca-cert\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.606727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606584    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-certs\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.606727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606627    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-data\") pod \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\" (UID: \"1ba6a000-311b-4f1f-97f6-298bb0e2f87e\") "
Nov 01 14:44:39.607077 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.606919    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istiod-ca-cert" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:39.607505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.607263    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-certs" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.607505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.607297    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-data" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.607505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.607438    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.607505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.607454    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.607505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.607464    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.607505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.607475    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.607505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.607489    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.608972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.608947    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-kube-api-access-jb9s2" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "kube-api-access-jb9s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:39.609055 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.608971    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-token" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:39.609152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.609114    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-podinfo" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:39.609269 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.609177    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-envoy" pod "1ba6a000-311b-4f1f-97f6-298bb0e2f87e" (UID: "1ba6a000-311b-4f1f-97f6-298bb0e2f87e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.610751 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.610738    5423 scope.go:122] "RemoveContainer" containerID="658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6"
Nov 01 14:44:39.610997 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:39.610972    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6\": container with ID starting with 658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6 not found: ID does not exist" containerID="658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6"
Nov 01 14:44:39.611076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.610996    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6"} err="failed to get container status \"658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6\": rpc error: code = NotFound desc = could not find container \"658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6\": container with ID starting with 658c479806d3f68b0df9ab5b3e7947de58614cd1935ce121894223316de0aed6 not found: ID does not exist"
Nov 01 14:44:39.614158 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.614138    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:44:39.616751 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.616734    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:44:39.708109 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708074    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-certs\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.708289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708141    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-socket\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.708289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708165    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904a66ff-67b2-4a86-8756-ceeb67668dff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904a66ff-67b2-4a86-8756-ceeb67668dff-istiod-ca-cert\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.708289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708189    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-credential-socket\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.708289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708206    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-socket\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.708289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708248    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82556296-1a82-4ac7-828c-b8a223aa3fd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82556296-1a82-4ac7-828c-b8a223aa3fd7-istiod-ca-cert\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.708289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708269    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-certs\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.708289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708288    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-data\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708335    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-podinfo\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708372    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-data\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708410    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-token\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-token\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708456    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-certs" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708470    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dda68f1-989b-4643-9013-0e2962c9b4d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dda68f1-989b-4643-9013-0e2962c9b4d9-istiod-ca-cert\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708541    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-kube-api-access-fhptm\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708579    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-kube-api-access-sghl9\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-kube-api-access-sghl9\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708657    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-token\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-token\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708701    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-podinfo\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.708722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708713    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dda68f1-989b-4643-9013-0e2962c9b4d9-istiod-ca-cert" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:39.709215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708735    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-certs\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.709215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708798    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-socket\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.709215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708845    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-kube-api-access-j978v\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-kube-api-access-j978v\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.709215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.708903    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-socket" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.709215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709126    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904a66ff-67b2-4a86-8756-ceeb67668dff-istiod-ca-cert" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:39.709215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709150    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-envoy\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.709215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709181    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-token\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-token\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.709648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709238    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-credential-socket\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.709648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709292    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-envoy\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.709648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709327    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-data\") pod \"82556296-1a82-4ac7-828c-b8a223aa3fd7\" (UID: \"82556296-1a82-4ac7-828c-b8a223aa3fd7\") "
Nov 01 14:44:39.709648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709370    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-podinfo\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.709648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709415    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-credential-socket\") pod \"7dda68f1-989b-4643-9013-0e2962c9b4d9\" (UID: \"7dda68f1-989b-4643-9013-0e2962c9b4d9\") "
Nov 01 14:44:39.709648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.709440    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-envoy\") pod \"904a66ff-67b2-4a86-8756-ceeb67668dff\" (UID: \"904a66ff-67b2-4a86-8756-ceeb67668dff\") "
Nov 01 14:44:39.710212 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710184    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb9s2\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-kube-api-access-jb9s2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.710332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710217    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.710332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710232    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.710332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710248    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904a66ff-67b2-4a86-8756-ceeb67668dff-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.710332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710263    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.710332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710276    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.710332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710290    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dda68f1-989b-4643-9013-0e2962c9b4d9-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.710332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710304    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba6a000-311b-4f1f-97f6-298bb0e2f87e-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.710780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.710530    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-certs" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.711458 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.711434    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-certs" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.711613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.711583    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-kube-api-access-j978v" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "kube-api-access-j978v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:39.711845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.711797    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-socket" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.711996 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.711974    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-credential-socket" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.712285 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.712264    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-socket" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.712730 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.712710    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82556296-1a82-4ac7-828c-b8a223aa3fd7-istiod-ca-cert" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:39.713104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.713079    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-data" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.713347 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.713325    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-token" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:39.713688 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.713666    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-data" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.713797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.713775    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-podinfo" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:39.714666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.714169    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-credential-socket" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.714666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.714366    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-data" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.715644 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.715617    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-kube-api-access-fhptm" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "kube-api-access-fhptm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:39.715869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.715848    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-credential-socket" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.715949 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.715883    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-envoy" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.715949 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.715897    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-kube-api-access-sghl9" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "kube-api-access-sghl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:39.717723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.717698    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-envoy" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.723871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.723806    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-podinfo" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:39.724203 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.724099    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-token" pod "904a66ff-67b2-4a86-8756-ceeb67668dff" (UID: "904a66ff-67b2-4a86-8756-ceeb67668dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:39.724203 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.724103    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-podinfo" pod "7dda68f1-989b-4643-9013-0e2962c9b4d9" (UID: "7dda68f1-989b-4643-9013-0e2962c9b4d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:39.724203 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.724174    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-token" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:39.725654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.725631    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-envoy" pod "82556296-1a82-4ac7-828c-b8a223aa3fd7" (UID: "82556296-1a82-4ac7-828c-b8a223aa3fd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:39.811629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811596    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811622    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811632    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811640    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811649    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811658    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811667    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811675    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811686    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82556296-1a82-4ac7-828c-b8a223aa3fd7-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811694    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811704    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811712    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811722    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811730    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811739    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhptm\" (UniqueName: \"kubernetes.io/projected/904a66ff-67b2-4a86-8756-ceeb67668dff-kube-api-access-fhptm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811749    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sghl9\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-kube-api-access-sghl9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811758    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dda68f1-989b-4643-9013-0e2962c9b4d9-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811766    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904a66ff-67b2-4a86-8756-ceeb67668dff-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811774    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811782    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811791    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j978v\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-kube-api-access-j978v\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811800    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:39.811857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:39.811807    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82556296-1a82-4ac7-828c-b8a223aa3fd7-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:40.307664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.307633    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"]
Nov 01 14:44:40.310659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.310639    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4sjlm"]
Nov 01 14:44:40.369704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.369675    5423 scope.go:122] "RemoveContainer" containerID="0f8fafbace9dbcef93425f99fe02b139e8db5478e9d0930cd7a5c632f113f6ec"
Nov 01 14:44:40.379316 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.379295    5423 scope.go:122] "RemoveContainer" containerID="c9259cdf042adca33c63acd0c8e6937b3b8dcf1494920d9afd61f66e65ec3350"
Nov 01 14:44:40.422236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.422196    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"]
Nov 01 14:44:40.425451 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.425433    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-zq4hf"]
Nov 01 14:44:40.586535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.586430    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh" event={"ID":"904a66ff-67b2-4a86-8756-ceeb67668dff","Type":"ContainerDied","Data":"a4b8c6f119a30120730d4a65edfbe2fbbadfcd31f998bd5a443360a455e70ede"}
Nov 01 14:44:40.586535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.586464    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"
Nov 01 14:44:40.586535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.586488    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"
Nov 01 14:44:40.586535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.586512    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2" event={"ID":"82556296-1a82-4ac7-828c-b8a223aa3fd7","Type":"ContainerDied","Data":"1a9a962f522ed3d934eab79b86645a7a3d6dbc04b6b20b93b555ddff9930c231"}
Nov 01 14:44:40.882632 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.882551    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"]
Nov 01 14:44:40.886657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.886630    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-bvdp2"]
Nov 01 14:44:40.998472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:40.998440    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"]
Nov 01 14:44:41.002182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:41.002159    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2pkvh"]
Nov 01 14:44:41.961717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:41.961679    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" path="/var/lib/kubelet/pods/1ba6a000-311b-4f1f-97f6-298bb0e2f87e/volumes"
Nov 01 14:44:41.962212 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:41.962147    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" path="/var/lib/kubelet/pods/7dda68f1-989b-4643-9013-0e2962c9b4d9/volumes"
Nov 01 14:44:41.962607 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:41.962592    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" path="/var/lib/kubelet/pods/82556296-1a82-4ac7-828c-b8a223aa3fd7/volumes"
Nov 01 14:44:41.963328 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:41.963311    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" path="/var/lib/kubelet/pods/904a66ff-67b2-4a86-8756-ceeb67668dff/volumes"
Nov 01 14:44:47.117117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117079    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"]
Nov 01 14:44:47.117903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117880    5423 cpu_manager.go:414] "containerMap: removing container" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy"
Nov 01 14:44:47.117995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117903    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy"
Nov 01 14:44:47.117995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117927    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy"
Nov 01 14:44:47.117995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117933    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy"
Nov 01 14:44:47.117995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117950    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e9a04f92-ec68-40e5-9a2b-39bdbf4919be" containerName="payload-processing"
Nov 01 14:44:47.117995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117958    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a04f92-ec68-40e5-9a2b-39bdbf4919be" containerName="payload-processing"
Nov 01 14:44:47.117995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117981    5423 cpu_manager.go:414] "containerMap: removing container" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy"
Nov 01 14:44:47.117995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.117988    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118007    5423 cpu_manager.go:414] "containerMap: removing container" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118014    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118027    5423 cpu_manager.go:414] "containerMap: removing container" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118035    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118044    5423 cpu_manager.go:414] "containerMap: removing container" podUID="ac7599a1-3847-46aa-874d-4a738de877e0" containerName="payload-pre-processing"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118052    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7599a1-3847-46aa-874d-4a738de877e0" containerName="payload-pre-processing"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118089    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118098    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118239    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="82556296-1a82-4ac7-828c-b8a223aa3fd7" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118254    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118264    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a04f92-ec68-40e5-9a2b-39bdbf4919be" containerName="payload-processing"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118275    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118285    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118294    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a3901c-5d4f-4378-9778-e4186b9863e3" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118304    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="904a66ff-67b2-4a86-8756-ceeb67668dff" containerName="istio-proxy"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118312    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7599a1-3847-46aa-874d-4a738de877e0" containerName="payload-pre-processing"
Nov 01 14:44:47.118381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.118326    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba6a000-311b-4f1f-97f6-298bb0e2f87e" containerName="istio-proxy"
Nov 01 14:44:47.123940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.123916    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:47.174806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.174780    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"]
Nov 01 14:44:47.218991 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.218962    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn\" (UID: \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:47.219147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.219125    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcpvz\" (UniqueName: \"kubernetes.io/projected/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-kube-api-access-pcpvz\") pod \"payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn\" (UID: \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:47.223629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.223611    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-18a879-dockercfg-9v257\""
Nov 01 14:44:47.223712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.223638    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-18a879\""
Nov 01 14:44:47.320796 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.320766    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn\" (UID: \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:47.320949 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.320866    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcpvz\" (UniqueName: \"kubernetes.io/projected/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-kube-api-access-pcpvz\") pod \"payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn\" (UID: \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:47.321511 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.321489    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn\" (UID: \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:47.379197 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.379129    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcpvz\" (UniqueName: \"kubernetes.io/projected/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-kube-api-access-pcpvz\") pod \"payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn\" (UID: \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:47.437677 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.437653    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:47.571079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.571044    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"]
Nov 01 14:44:47.571974 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.571948    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy"
Nov 01 14:44:47.571974 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.571973    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dda68f1-989b-4643-9013-0e2962c9b4d9" containerName="istio-proxy"
Nov 01 14:44:47.577318 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.577294    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:47.624403 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.624355    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpv6\" (UniqueName: \"kubernetes.io/projected/09d974fc-18ae-41ee-8bac-5e22365f1cd6-kube-api-access-rdpv6\") pod \"payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf\" (UID: \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:47.624558 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.624469    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d974fc-18ae-41ee-8bac-5e22365f1cd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf\" (UID: \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:47.627139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.627113    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"]
Nov 01 14:44:47.726136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.726054    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpv6\" (UniqueName: \"kubernetes.io/projected/09d974fc-18ae-41ee-8bac-5e22365f1cd6-kube-api-access-rdpv6\") pod \"payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf\" (UID: \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:47.726136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.726107    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d974fc-18ae-41ee-8bac-5e22365f1cd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf\" (UID: \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:47.726819 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.726794    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d974fc-18ae-41ee-8bac-5e22365f1cd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf\" (UID: \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:47.783810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.783782    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpv6\" (UniqueName: \"kubernetes.io/projected/09d974fc-18ae-41ee-8bac-5e22365f1cd6-kube-api-access-rdpv6\") pod \"payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf\" (UID: \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:47.802815 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:44:47.802785    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-3d347fba94ee8bd58f4b68830ee6d0b75186a2dce0eb00de64d1516f0b18b856 WatchSource:0}: Error finding container 3d347fba94ee8bd58f4b68830ee6d0b75186a2dce0eb00de64d1516f0b18b856: Status 404 returned error can't find the container with id 3d347fba94ee8bd58f4b68830ee6d0b75186a2dce0eb00de64d1516f0b18b856
Nov 01 14:44:47.805008 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.804984    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:44:47.805546 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.805527    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"]
Nov 01 14:44:47.896512 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:47.896484    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:48.262235 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:44:48.262203    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d974fc_18ae_41ee_8bac_5e22365f1cd6.slice/crio-e30684ce58bff2554ef71abd2bf8b2429b73e03c816dd12d93b05dc1169042de WatchSource:0}: Error finding container e30684ce58bff2554ef71abd2bf8b2429b73e03c816dd12d93b05dc1169042de: Status 404 returned error can't find the container with id e30684ce58bff2554ef71abd2bf8b2429b73e03c816dd12d93b05dc1169042de
Nov 01 14:44:48.265327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:48.265306    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"]
Nov 01 14:44:48.636304 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:48.636268    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf" event={"ID":"09d974fc-18ae-41ee-8bac-5e22365f1cd6","Type":"ContainerStarted","Data":"e30684ce58bff2554ef71abd2bf8b2429b73e03c816dd12d93b05dc1169042de"}
Nov 01 14:44:48.637590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:48.637562    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn" event={"ID":"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0","Type":"ContainerStarted","Data":"3d347fba94ee8bd58f4b68830ee6d0b75186a2dce0eb00de64d1516f0b18b856"}
Nov 01 14:44:50.658629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:50.658593    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn" event={"ID":"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0","Type":"ContainerStarted","Data":"f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19"}
Nov 01 14:44:50.659457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:50.659434    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:44:50.778698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:50.778648    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn" podStartSLOduration=1.198015875 podStartE2EDuration="3.778634237s" totalImagesPullingTime="2.580618362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:50.775638155 +0000 UTC m=+1630.941123454" watchObservedRunningTime="2026-11-01 14:44:50.778634237 +0000 UTC m=+1630.944119539"
Nov 01 14:44:51.666101 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:51.666058    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf" event={"ID":"09d974fc-18ae-41ee-8bac-5e22365f1cd6","Type":"ContainerStarted","Data":"83cb4fd12b66107b4efd4b22155e8fe8db401476faed771faba1afb4d88f7f4e"}
Nov 01 14:44:51.666653 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:51.666405    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:44:51.783337 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:51.783284    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf" podStartSLOduration=2.437546168 podStartE2EDuration="4.78326768s" totalImagesPullingTime="2.345721512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:51.780772335 +0000 UTC m=+1631.946257633" watchObservedRunningTime="2026-11-01 14:44:51.78326768 +0000 UTC m=+1631.948753006"
Nov 01 14:44:53.120144 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:53.120106    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"]
Nov 01 14:44:53.120620 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:53.120350    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" containerID="cri-o://ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3" gracePeriod=30
Nov 01 14:44:53.171326 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:53.171297    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"]
Nov 01 14:44:53.171525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:53.171501    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" containerID="cri-o://7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec" gracePeriod=30
Nov 01 14:44:53.173667 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:53.173644    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"]
Nov 01 14:44:53.173844 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:53.173821    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy" containerID="cri-o://fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c" gracePeriod=30
Nov 01 14:44:57.300885 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.300846    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-j2pxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:44:57.301235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.300907    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:44:57.354094 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.354057    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8dh5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:44:57.354246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.354125    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:44:57.740469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.740438    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"]
Nov 01 14:44:57.744696 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.744677    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.801542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.801513    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-25ce0b-serving-cert\""
Nov 01 14:44:57.804536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.804513    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"]
Nov 01 14:44:57.806268 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.806245    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b391c51-7998-423f-bc7d-412cfd17f2a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.806415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.806313    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.806483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.806464    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghxpv\" (UniqueName: \"kubernetes.io/projected/9b391c51-7998-423f-bc7d-412cfd17f2a0-kube-api-access-ghxpv\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.806538 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.806499    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-metrics-certs\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.809784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.809752    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-drf59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:44:57.809880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.809833    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:44:57.907292 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.907261    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.907449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.907411    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghxpv\" (UniqueName: \"kubernetes.io/projected/9b391c51-7998-423f-bc7d-412cfd17f2a0-kube-api-access-ghxpv\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.907449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.907440    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-metrics-certs\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.907554 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:57.907449    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-serving-cert: secret "maas-api-e2e-ipp-a-25ce0b-serving-cert" not found
Nov 01 14:44:57.907554 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:57.907529    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls podName:9b391c51-7998-423f-bc7d-412cfd17f2a0 nodeName:}" failed. No retries permitted until 2026-11-01 14:44:58.407505684 +0000 UTC m=+1638.572990976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls") pod "maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x" (UID: "9b391c51-7998-423f-bc7d-412cfd17f2a0") : secret "maas-api-e2e-ipp-a-25ce0b-serving-cert" not found
Nov 01 14:44:57.907637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.907622    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b391c51-7998-423f-bc7d-412cfd17f2a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.908259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.908235    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b391c51-7998-423f-bc7d-412cfd17f2a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.909965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.909944    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-metrics-certs\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:57.966702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:57.966670    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghxpv\" (UniqueName: \"kubernetes.io/projected/9b391c51-7998-423f-bc7d-412cfd17f2a0-kube-api-access-ghxpv\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:58.421061 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.421024    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:58.424292 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.424270    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:58.456741 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.456714    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:44:58.495761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.495736    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:44:58.510503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.510480    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:44:58.521565 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521538    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-token\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-token\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.521706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521583    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-data\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.521706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521617    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-credential-socket\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.521706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521643    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-envoy\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.521706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521694    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b73d126-74d5-4c66-8e08-ad5a253f509d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b73d126-74d5-4c66-8e08-ad5a253f509d-istiod-ca-cert\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.521927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521726    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-certs\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.521927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521756    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-podinfo\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.521927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521801    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/260e919a-8650-4d1f-8023-9f41eaba3278-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260e919a-8650-4d1f-8023-9f41eaba3278-istio-podinfo\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.521927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521831    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-data\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.521927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521843    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-data" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.521927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521863    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-socket\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521930    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-kube-api-access-x2f92\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-kube-api-access-x2f92\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.521965    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-kube-api-access-gt5hj\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-kube-api-access-gt5hj\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522031    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-data\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522076    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-certs\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522111    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-credential-socket\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522143    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istiod-ca-cert\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522007    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-certs" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522176    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-socket\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.522258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522239    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-socket\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522260    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b73d126-74d5-4c66-8e08-ad5a253f509d-istiod-ca-cert" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522343    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-token\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-token\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522410    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-socket" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522380    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-istio-token\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-istio-token\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522499    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-podinfo\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522535    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-certs\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522570    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-credential-socket\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522602    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-kube-api-access-zk4v9\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-kube-api-access-zk4v9\") pod \"6b73d126-74d5-4c66-8e08-ad5a253f509d\" (UID: \"6b73d126-74d5-4c66-8e08-ad5a253f509d\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522649    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260e919a-8650-4d1f-8023-9f41eaba3278-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260e919a-8650-4d1f-8023-9f41eaba3278-istiod-ca-cert\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522681    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-envoy\") pod \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\" (UID: \"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9\") "
Nov 01 14:44:58.522750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522709    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-credential-socket" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.523185 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.522771    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-envoy\") pod \"260e919a-8650-4d1f-8023-9f41eaba3278\" (UID: \"260e919a-8650-4d1f-8023-9f41eaba3278\") "
Nov 01 14:44:58.524455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.523376    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-socket" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.524627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.524607    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-data" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.524690 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.524620    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/260e919a-8650-4d1f-8023-9f41eaba3278-istio-podinfo" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:58.524690 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.523528    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-data" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.524867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.524844    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-certs" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.525020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.524987    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-credential-socket" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.525091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.525047    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-podinfo" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:58.525460 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.525208    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260e919a-8650-4d1f-8023-9f41eaba3278-istiod-ca-cert" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:58.525910 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.525692    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istiod-ca-cert" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:58.526553 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526529    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526557    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260e919a-8650-4d1f-8023-9f41eaba3278-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526573    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526590    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526610    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b73d126-74d5-4c66-8e08-ad5a253f509d-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526625    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526704    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526723    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260e919a-8650-4d1f-8023-9f41eaba3278-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526738    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526762    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.526858 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526776    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.527148 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526885    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.527148 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526907    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.527148 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.526923    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.527348 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.527322    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-istio-token" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:58.527571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.527372    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-token" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:58.527571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.527533    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-certs" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.529637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.529610    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-credential-socket" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.529737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.529671    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-socket" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.529800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.529765    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-kube-api-access-zk4v9" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "kube-api-access-zk4v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:58.529851 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.529835    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-token" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:58.529938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.529916    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-envoy" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.531016 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.530952    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-envoy" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.531016 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.530967    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-envoy" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:58.531287 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.531262    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-kube-api-access-gt5hj" pod "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" (UID: "826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9"). InnerVolumeSpecName "kube-api-access-gt5hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:58.531420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.531354    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-podinfo" pod "6b73d126-74d5-4c66-8e08-ad5a253f509d" (UID: "6b73d126-74d5-4c66-8e08-ad5a253f509d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:58.534424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.532618    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-kube-api-access-x2f92" pod "260e919a-8650-4d1f-8023-9f41eaba3278" (UID: "260e919a-8650-4d1f-8023-9f41eaba3278"). InnerVolumeSpecName "kube-api-access-x2f92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:58.628096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628060    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628087    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628097    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628106    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628116    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2f92\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-kube-api-access-x2f92\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628127    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5hj\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-kube-api-access-gt5hj\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628149    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260e919a-8650-4d1f-8023-9f41eaba3278-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628158    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628166    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260e919a-8650-4d1f-8023-9f41eaba3278-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628174    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b73d126-74d5-4c66-8e08-ad5a253f509d-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628182    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b73d126-74d5-4c66-8e08-ad5a253f509d-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628191    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4v9\" (UniqueName: \"kubernetes.io/projected/6b73d126-74d5-4c66-8e08-ad5a253f509d-kube-api-access-zk4v9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.628407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.628199    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:58.659115 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.659084    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:44:58.723822 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.723744    5423 generic.go:356] "Generic (PLEG): container finished" podID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerID="ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3" exitCode=0
Nov 01 14:44:58.723955 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.723848    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" event={"ID":"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9","Type":"ContainerDied","Data":"ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3"}
Nov 01 14:44:58.723955 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.723936    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz" event={"ID":"826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9","Type":"ContainerDied","Data":"a705e38777f000dfd420b301b870337c3427d74e445573b4f1c2435abf6c5ca1"}
Nov 01 14:44:58.724023 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.723970    5423 scope.go:122] "RemoveContainer" containerID="ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3"
Nov 01 14:44:58.724167 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.724147    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"
Nov 01 14:44:58.738925 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.738889    5423 generic.go:356] "Generic (PLEG): container finished" podID="260e919a-8650-4d1f-8023-9f41eaba3278" containerID="fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c" exitCode=0
Nov 01 14:44:58.739061 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.739038    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" event={"ID":"260e919a-8650-4d1f-8023-9f41eaba3278","Type":"ContainerDied","Data":"fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c"}
Nov 01 14:44:58.739125 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.739078    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59" event={"ID":"260e919a-8650-4d1f-8023-9f41eaba3278","Type":"ContainerDied","Data":"3e8789bb912c550e44087386928a452cd52bd2e2ad9cc43712350c65cf40e9b1"}
Nov 01 14:44:58.739184 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.739157    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"
Nov 01 14:44:58.741890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.741757    5423 generic.go:356] "Generic (PLEG): container finished" podID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerID="7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec" exitCode=0
Nov 01 14:44:58.742800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.742749    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" event={"ID":"6b73d126-74d5-4c66-8e08-ad5a253f509d","Type":"ContainerDied","Data":"7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec"}
Nov 01 14:44:58.742910 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.742845    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f" event={"ID":"6b73d126-74d5-4c66-8e08-ad5a253f509d","Type":"ContainerDied","Data":"9512fecda228bafdeadf85ef54635b345aa3389ffb86a83342feb6294f5bffab"}
Nov 01 14:44:58.742995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.742981    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"
Nov 01 14:44:58.754365 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.754346    5423 scope.go:122] "RemoveContainer" containerID="ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3"
Nov 01 14:44:58.754769 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:58.754739    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3\": container with ID starting with ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3 not found: ID does not exist" containerID="ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3"
Nov 01 14:44:58.754864 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.754781    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3"} err="failed to get container status \"ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3\": rpc error: code = NotFound desc = could not find container \"ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3\": container with ID starting with ebd7b20fa44013a4f6f8f8f63bfd41322fa5c9bad97a2ad1367e10260fd3fdf3 not found: ID does not exist"
Nov 01 14:44:58.754864 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.754807    5423 scope.go:122] "RemoveContainer" containerID="fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c"
Nov 01 14:44:58.766721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.766702    5423 scope.go:122] "RemoveContainer" containerID="fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c"
Nov 01 14:44:58.769157 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:58.769133    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c\": container with ID starting with fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c not found: ID does not exist" containerID="fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c"
Nov 01 14:44:58.769270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.769168    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c"} err="failed to get container status \"fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c\": rpc error: code = NotFound desc = could not find container \"fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c\": container with ID starting with fc1075608fdb1f33a15edea2fff12bb0c37b2d7934cff90bf2bf101d72c1055c not found: ID does not exist"
Nov 01 14:44:58.769270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.769189    5423 scope.go:122] "RemoveContainer" containerID="7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec"
Nov 01 14:44:58.784652 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.780601    5423 scope.go:122] "RemoveContainer" containerID="7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec"
Nov 01 14:44:58.787986 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:44:58.786217    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec\": container with ID starting with 7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec not found: ID does not exist" containerID="7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec"
Nov 01 14:44:58.787986 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.786424    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec"} err="failed to get container status \"7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec\": rpc error: code = NotFound desc = could not find container \"7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec\": container with ID starting with 7b5a0d70fbe6a667fb0b9c677b6d84fa820f4039301ba8d59da6a9a789ac53ec not found: ID does not exist"
Nov 01 14:44:58.939244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.939208    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"]
Nov 01 14:44:58.941620 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:58.941598    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-j2pxz"]
Nov 01 14:44:59.050447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.050404    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"]
Nov 01 14:44:59.050657 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:44:59.050635    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b391c51_7998_423f_bc7d_412cfd17f2a0.slice/crio-9f9803580fad1e543ecb0b90e30445c678f6dbc102b0453403952a4a13ab9fd8 WatchSource:0}: Error finding container 9f9803580fad1e543ecb0b90e30445c678f6dbc102b0453403952a4a13ab9fd8: Status 404 returned error can't find the container with id 9f9803580fad1e543ecb0b90e30445c678f6dbc102b0453403952a4a13ab9fd8
Nov 01 14:44:59.053112 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.053094    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"]
Nov 01 14:44:59.062715 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.062693    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8dh5f"]
Nov 01 14:44:59.174072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.174040    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"]
Nov 01 14:44:59.177586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.177560    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-drf59"]
Nov 01 14:44:59.466984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.466950    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7387-1f0b13"]
Nov 01 14:44:59.467558 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467542    5423 cpu_manager.go:414] "containerMap: removing container" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy"
Nov 01 14:44:59.467558 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467559    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy"
Nov 01 14:44:59.467634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467574    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy"
Nov 01 14:44:59.467634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467580    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy"
Nov 01 14:44:59.467634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467599    5423 cpu_manager.go:414] "containerMap: removing container" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy"
Nov 01 14:44:59.467634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467604    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy"
Nov 01 14:44:59.467767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467683    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" containerName="istio-proxy"
Nov 01 14:44:59.467767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467698    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" containerName="istio-proxy"
Nov 01 14:44:59.467767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.467706    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" containerName="istio-proxy"
Nov 01 14:44:59.471371 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.471350    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:44:59.519881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.519851    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-1f0b13"]
Nov 01 14:44:59.524965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.524939    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jngw\""
Nov 01 14:44:59.543892 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.543869    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6g2j\" (UniqueName: \"kubernetes.io/projected/fab67ded-c1af-46c3-8530-e426b088c1ec-kube-api-access-g6g2j\") pod \"test-curl-7387-1f0b13\" (UID: \"fab67ded-c1af-46c3-8530-e426b088c1ec\") " pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:44:59.644901 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.644873    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6g2j\" (UniqueName: \"kubernetes.io/projected/fab67ded-c1af-46c3-8530-e426b088c1ec-kube-api-access-g6g2j\") pod \"test-curl-7387-1f0b13\" (UID: \"fab67ded-c1af-46c3-8530-e426b088c1ec\") " pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:44:59.702846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.702819    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6g2j\" (UniqueName: \"kubernetes.io/projected/fab67ded-c1af-46c3-8530-e426b088c1ec-kube-api-access-g6g2j\") pod \"test-curl-7387-1f0b13\" (UID: \"fab67ded-c1af-46c3-8530-e426b088c1ec\") " pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:44:59.751919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.751836    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x" event={"ID":"9b391c51-7998-423f-bc7d-412cfd17f2a0","Type":"ContainerStarted","Data":"9f9803580fad1e543ecb0b90e30445c678f6dbc102b0453403952a4a13ab9fd8"}
Nov 01 14:44:59.784533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.784507    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:44:59.961299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.961277    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260e919a-8650-4d1f-8023-9f41eaba3278" path="/var/lib/kubelet/pods/260e919a-8650-4d1f-8023-9f41eaba3278/volumes"
Nov 01 14:44:59.961797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.961783    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b73d126-74d5-4c66-8e08-ad5a253f509d" path="/var/lib/kubelet/pods/6b73d126-74d5-4c66-8e08-ad5a253f509d/volumes"
Nov 01 14:44:59.962208 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:44:59.962196    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9" path="/var/lib/kubelet/pods/826dbf37-aeb7-4cfe-9a0f-f98406cbcfd9/volumes"
Nov 01 14:45:00.153234 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:45:00.153204    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab67ded_c1af_46c3_8530_e426b088c1ec.slice/crio-4ea24c83105f0bb6ff0e70de9fc28e0c3f4381e68a20481c5fc75f529da4ea26 WatchSource:0}: Error finding container 4ea24c83105f0bb6ff0e70de9fc28e0c3f4381e68a20481c5fc75f529da4ea26: Status 404 returned error can't find the container with id 4ea24c83105f0bb6ff0e70de9fc28e0c3f4381e68a20481c5fc75f529da4ea26
Nov 01 14:45:00.156413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.156378    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-1f0b13"]
Nov 01 14:45:00.185524 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.185499    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"]
Nov 01 14:45:00.191300 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.191281    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"]
Nov 01 14:45:00.191426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.191410    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:00.195225 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.195201    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:00.204300 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.204280    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"]
Nov 01 14:45:00.208494 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.208156    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:00.250071 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.245959    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 14:45:00.269869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.269847    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5a0187-9e1e-4b8c-9caa-e505814159ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw\" (UID: \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:00.269977 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.269917    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955b737-485f-42e8-beb3-cda54d0d64a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4\" (UID: \"2955b737-485f-42e8-beb3-cda54d0d64a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:00.269977 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.269943    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8225f9f-a744-4fd9-9678-9b85091004b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp\" (UID: \"b8225f9f-a744-4fd9-9678-9b85091004b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:00.270048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.269980    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kvh7\" (UniqueName: \"kubernetes.io/projected/1b5a0187-9e1e-4b8c-9caa-e505814159ab-kube-api-access-4kvh7\") pod \"maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw\" (UID: \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:00.270098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.270078    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dntv\" (UniqueName: \"kubernetes.io/projected/2955b737-485f-42e8-beb3-cda54d0d64a8-kube-api-access-5dntv\") pod \"maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4\" (UID: \"2955b737-485f-42e8-beb3-cda54d0d64a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:00.270147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.270120    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8rf2\" (UniqueName: \"kubernetes.io/projected/b8225f9f-a744-4fd9-9678-9b85091004b5-kube-api-access-l8rf2\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp\" (UID: \"b8225f9f-a744-4fd9-9678-9b85091004b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:00.303153 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.303122    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"]
Nov 01 14:45:00.347883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.347763    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"]
Nov 01 14:45:00.370873 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.370844    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kvh7\" (UniqueName: \"kubernetes.io/projected/1b5a0187-9e1e-4b8c-9caa-e505814159ab-kube-api-access-4kvh7\") pod \"maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw\" (UID: \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:00.371025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.370892    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dntv\" (UniqueName: \"kubernetes.io/projected/2955b737-485f-42e8-beb3-cda54d0d64a8-kube-api-access-5dntv\") pod \"maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4\" (UID: \"2955b737-485f-42e8-beb3-cda54d0d64a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:00.371025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.370937    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8rf2\" (UniqueName: \"kubernetes.io/projected/b8225f9f-a744-4fd9-9678-9b85091004b5-kube-api-access-l8rf2\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp\" (UID: \"b8225f9f-a744-4fd9-9678-9b85091004b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:00.371025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.371016    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5a0187-9e1e-4b8c-9caa-e505814159ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw\" (UID: \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:00.371229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.371087    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955b737-485f-42e8-beb3-cda54d0d64a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4\" (UID: \"2955b737-485f-42e8-beb3-cda54d0d64a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:00.371229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.371117    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8225f9f-a744-4fd9-9678-9b85091004b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp\" (UID: \"b8225f9f-a744-4fd9-9678-9b85091004b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:00.371822 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.371799    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5a0187-9e1e-4b8c-9caa-e505814159ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw\" (UID: \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:00.371926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.371823    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8225f9f-a744-4fd9-9678-9b85091004b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp\" (UID: \"b8225f9f-a744-4fd9-9678-9b85091004b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:00.371926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.371824    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955b737-485f-42e8-beb3-cda54d0d64a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4\" (UID: \"2955b737-485f-42e8-beb3-cda54d0d64a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:00.407618 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.407546    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"]
Nov 01 14:45:00.429638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.429609    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kvh7\" (UniqueName: \"kubernetes.io/projected/1b5a0187-9e1e-4b8c-9caa-e505814159ab-kube-api-access-4kvh7\") pod \"maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw\" (UID: \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:00.429738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.429710    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dntv\" (UniqueName: \"kubernetes.io/projected/2955b737-485f-42e8-beb3-cda54d0d64a8-kube-api-access-5dntv\") pod \"maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4\" (UID: \"2955b737-485f-42e8-beb3-cda54d0d64a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:00.431149 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.431129    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8rf2\" (UniqueName: \"kubernetes.io/projected/b8225f9f-a744-4fd9-9678-9b85091004b5-kube-api-access-l8rf2\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp\" (UID: \"b8225f9f-a744-4fd9-9678-9b85091004b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:00.516204 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.516172    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:00.524585 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.524561    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:00.530414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.530334    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:00.760302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.760203    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-1f0b13" event={"ID":"fab67ded-c1af-46c3-8530-e426b088c1ec","Type":"ContainerStarted","Data":"d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9"}
Nov 01 14:45:00.760302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.760252    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-1f0b13" event={"ID":"fab67ded-c1af-46c3-8530-e426b088c1ec","Type":"ContainerStarted","Data":"4ea24c83105f0bb6ff0e70de9fc28e0c3f4381e68a20481c5fc75f529da4ea26"}
Nov 01 14:45:00.761781 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.761756    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x" event={"ID":"9b391c51-7998-423f-bc7d-412cfd17f2a0","Type":"ContainerStarted","Data":"dd421d26859f6c6ca58f16b2d5a0b3b513d149fbe93e8665c52b5b44b29198a8"}
Nov 01 14:45:00.761920 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.761902    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:45:00.884267 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.884205    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7387-1f0b13" podStartSLOduration=1.8841913 podStartE2EDuration="1.8841913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:00.883832971 +0000 UTC m=+1641.049318270" watchObservedRunningTime="2026-11-01 14:45:00.8841913 +0000 UTC m=+1641.049676597"
Nov 01 14:45:00.961328 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:45:00.961295    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8225f9f_a744_4fd9_9678_9b85091004b5.slice/crio-e2d42013fec854b1c93fa02bedc0c76a53b92673bcf951137a6603a5fe4194d9 WatchSource:0}: Error finding container e2d42013fec854b1c93fa02bedc0c76a53b92673bcf951137a6603a5fe4194d9: Status 404 returned error can't find the container with id e2d42013fec854b1c93fa02bedc0c76a53b92673bcf951137a6603a5fe4194d9
Nov 01 14:45:00.965638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:00.965611    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"]
Nov 01 14:45:01.004304 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.004244    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x" podStartSLOduration=3.124873426 podStartE2EDuration="4.004229278s" totalImagesPullingTime="879.355852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:01.003854303 +0000 UTC m=+1641.169339603" watchObservedRunningTime="2026-11-01 14:45:01.004229278 +0000 UTC m=+1641.169714575"
Nov 01 14:45:01.009892 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:45:01.009867    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2955b737_485f_42e8_beb3_cda54d0d64a8.slice/crio-47c170d9d80d2deeea2258a3288711ba22c7ad93010b7593ad36d59111802969 WatchSource:0}: Error finding container 47c170d9d80d2deeea2258a3288711ba22c7ad93010b7593ad36d59111802969: Status 404 returned error can't find the container with id 47c170d9d80d2deeea2258a3288711ba22c7ad93010b7593ad36d59111802969
Nov 01 14:45:01.011679 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:45:01.011656    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5a0187_9e1e_4b8c_9caa_e505814159ab.slice/crio-8cd5c532ae1e676446f0cee8c24f09f023f1d8c22d44b5cc22ae462ee6f202d6 WatchSource:0}: Error finding container 8cd5c532ae1e676446f0cee8c24f09f023f1d8c22d44b5cc22ae462ee6f202d6: Status 404 returned error can't find the container with id 8cd5c532ae1e676446f0cee8c24f09f023f1d8c22d44b5cc22ae462ee6f202d6
Nov 01 14:45:01.011871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.011856    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"]
Nov 01 14:45:01.014868 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.014820    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"]
Nov 01 14:45:01.768941 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.768906    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" event={"ID":"b8225f9f-a744-4fd9-9678-9b85091004b5","Type":"ContainerStarted","Data":"927b72b710f849ae0a086090e0a6e90fcf48d6cf414f473f32a5ac84e791460d"}
Nov 01 14:45:01.769370 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.768968    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" event={"ID":"b8225f9f-a744-4fd9-9678-9b85091004b5","Type":"ContainerStarted","Data":"e2d42013fec854b1c93fa02bedc0c76a53b92673bcf951137a6603a5fe4194d9"}
Nov 01 14:45:01.770575 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.770546    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" event={"ID":"1b5a0187-9e1e-4b8c-9caa-e505814159ab","Type":"ContainerStarted","Data":"f09b335b2496dd5ed6c2afb09c85e27190ef2c556975899f5ce3632d72648497"}
Nov 01 14:45:01.770712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.770594    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" event={"ID":"1b5a0187-9e1e-4b8c-9caa-e505814159ab","Type":"ContainerStarted","Data":"8cd5c532ae1e676446f0cee8c24f09f023f1d8c22d44b5cc22ae462ee6f202d6"}
Nov 01 14:45:01.772229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.772205    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" event={"ID":"2955b737-485f-42e8-beb3-cda54d0d64a8","Type":"ContainerStarted","Data":"8999b99690afa341da3af5fd54a9f5e62e8434ffea0626c95c62873c714d514e"}
Nov 01 14:45:01.772341 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.772247    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" event={"ID":"2955b737-485f-42e8-beb3-cda54d0d64a8","Type":"ContainerStarted","Data":"47c170d9d80d2deeea2258a3288711ba22c7ad93010b7593ad36d59111802969"}
Nov 01 14:45:01.887330 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:01.887277    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" podStartSLOduration=1.887263395 podStartE2EDuration="1.887263395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:01.885603329 +0000 UTC m=+1642.051088627" watchObservedRunningTime="2026-11-01 14:45:01.887263395 +0000 UTC m=+1642.052748693"
Nov 01 14:45:02.031463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.030579    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" podStartSLOduration=2.030558983 podStartE2EDuration="2.030558983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:02.011490063 +0000 UTC m=+1642.176975362" watchObservedRunningTime="2026-11-01 14:45:02.030558983 +0000 UTC m=+1642.196044281"
Nov 01 14:45:02.161836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.161787    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" podStartSLOduration=2.161772098 podStartE2EDuration="2.161772098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:02.159095886 +0000 UTC m=+1642.324581184" watchObservedRunningTime="2026-11-01 14:45:02.161772098 +0000 UTC m=+1642.327257397"
Nov 01 14:45:02.231399 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.231344    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:45:02.332228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.332193    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7387-1f0b13"]
Nov 01 14:45:02.335910 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.335880    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7387-1f0b13"]
Nov 01 14:45:02.659273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.659190    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"]
Nov 01 14:45:02.659790 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.659776    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fab67ded-c1af-46c3-8530-e426b088c1ec" containerName="test-curl-7387-1f0b13"
Nov 01 14:45:02.659838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.659791    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab67ded-c1af-46c3-8530-e426b088c1ec" containerName="test-curl-7387-1f0b13"
Nov 01 14:45:02.659930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.659921    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab67ded-c1af-46c3-8530-e426b088c1ec" containerName="test-curl-7387-1f0b13"
Nov 01 14:45:02.663459 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.663443    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.701031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.701001    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:45:02.710409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.710009    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:45:02.722467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.719274    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"]
Nov 01 14:45:02.722467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.720320    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cbac83-serving-cert\""
Nov 01 14:45:02.864408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.860200    5423 generic.go:356] "Generic (PLEG): container finished" podID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerID="f09b335b2496dd5ed6c2afb09c85e27190ef2c556975899f5ce3632d72648497" exitCode=60
Nov 01 14:45:02.864408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.860295    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" event={"ID":"1b5a0187-9e1e-4b8c-9caa-e505814159ab","Type":"ContainerDied","Data":"f09b335b2496dd5ed6c2afb09c85e27190ef2c556975899f5ce3632d72648497"}
Nov 01 14:45:02.864408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.860612    5423 scope.go:122] "RemoveContainer" containerID="f09b335b2496dd5ed6c2afb09c85e27190ef2c556975899f5ce3632d72648497"
Nov 01 14:45:02.864408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.861503    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxms\" (UniqueName: \"kubernetes.io/projected/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-kube-api-access-vxxms\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.864408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.861602    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.864408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.861723    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.864408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.861766    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.867631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.866286    5423 generic.go:356] "Generic (PLEG): container finished" podID="2955b737-485f-42e8-beb3-cda54d0d64a8" containerID="8999b99690afa341da3af5fd54a9f5e62e8434ffea0626c95c62873c714d514e" exitCode=60
Nov 01 14:45:02.867631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.866576    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" event={"ID":"2955b737-485f-42e8-beb3-cda54d0d64a8","Type":"ContainerDied","Data":"8999b99690afa341da3af5fd54a9f5e62e8434ffea0626c95c62873c714d514e"}
Nov 01 14:45:02.867631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.866921    5423 scope.go:122] "RemoveContainer" containerID="8999b99690afa341da3af5fd54a9f5e62e8434ffea0626c95c62873c714d514e"
Nov 01 14:45:02.870886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.869550    5423 generic.go:356] "Generic (PLEG): container finished" podID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerID="927b72b710f849ae0a086090e0a6e90fcf48d6cf414f473f32a5ac84e791460d" exitCode=60
Nov 01 14:45:02.870886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.869674    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7387-1f0b13" podUID="fab67ded-c1af-46c3-8530-e426b088c1ec" containerName="test-curl-7387-1f0b13" containerID="cri-o://d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9" gracePeriod=2
Nov 01 14:45:02.870886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.870068    5423 scope.go:122] "RemoveContainer" containerID="927b72b710f849ae0a086090e0a6e90fcf48d6cf414f473f32a5ac84e791460d"
Nov 01 14:45:02.870886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.870346    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" event={"ID":"b8225f9f-a744-4fd9-9678-9b85091004b5","Type":"ContainerDied","Data":"927b72b710f849ae0a086090e0a6e90fcf48d6cf414f473f32a5ac84e791460d"}
Nov 01 14:45:02.984931 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.981848    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.984931 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.981951    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxms\" (UniqueName: \"kubernetes.io/projected/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-kube-api-access-vxxms\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.984931 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.982037    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.984931 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.982126    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.984931 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:02.984541    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:02.984931 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:02.984650    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-serving-cert: secret "maas-api-e2e-ipp-b-cbac83-serving-cert" not found
Nov 01 14:45:02.984931 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:02.984708    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls podName:1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9 nodeName:}" failed. No retries permitted until 2026-11-01 14:45:03.48469006 +0000 UTC m=+1643.650175337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls") pod "maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt" (UID: "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9") : secret "maas-api-e2e-ipp-b-cbac83-serving-cert" not found
Nov 01 14:45:03.000562 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.000503    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:03.129828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.129789    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxms\" (UniqueName: \"kubernetes.io/projected/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-kube-api-access-vxxms\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:03.183632 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:03.183530    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-1f0b13\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="fab67ded-c1af-46c3-8530-e426b088c1ec" pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:45:03.489724 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.489642    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:03.492076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.492051    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:03.577400 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.577355    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:03.878216 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.878184    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" event={"ID":"b8225f9f-a744-4fd9-9678-9b85091004b5","Type":"ContainerStarted","Data":"b9dd71be2e50c8b1dd95d0221e13c57f7ca5e99016a1fd643f66de29c2e25891"}
Nov 01 14:45:03.880139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.880112    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" event={"ID":"1b5a0187-9e1e-4b8c-9caa-e505814159ab","Type":"ContainerStarted","Data":"7d2d51806a6f87ab56c15390de3de122964a782412f66204b4912ce11efb2bd4"}
Nov 01 14:45:03.882028 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.882005    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" event={"ID":"2955b737-485f-42e8-beb3-cda54d0d64a8","Type":"ContainerStarted","Data":"33673c3c974cd82f25faf1b11260f0c07d7f345f6ff1a9dea7025c3bcb0c20bc"}
Nov 01 14:45:03.930595 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:03.930558    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-1f0b13\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="fab67ded-c1af-46c3-8530-e426b088c1ec" pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:45:03.960573 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:03.960550    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"]
Nov 01 14:45:04.006160 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:45:04.006132    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfac50d_0c1a_4bdb_bf9e_4c7955df34b9.slice/crio-b5f07778811abb6c4411176acbe9dac20e0757b525df1976c6f24e778309bb50 WatchSource:0}: Error finding container b5f07778811abb6c4411176acbe9dac20e0757b525df1976c6f24e778309bb50: Status 404 returned error can't find the container with id b5f07778811abb6c4411176acbe9dac20e0757b525df1976c6f24e778309bb50
Nov 01 14:45:04.890157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.890047    5423 generic.go:356] "Generic (PLEG): container finished" podID="2955b737-485f-42e8-beb3-cda54d0d64a8" containerID="33673c3c974cd82f25faf1b11260f0c07d7f345f6ff1a9dea7025c3bcb0c20bc" exitCode=60
Nov 01 14:45:04.890616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.890176    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" event={"ID":"2955b737-485f-42e8-beb3-cda54d0d64a8","Type":"ContainerDied","Data":"33673c3c974cd82f25faf1b11260f0c07d7f345f6ff1a9dea7025c3bcb0c20bc"}
Nov 01 14:45:04.890616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.890238    5423 scope.go:122] "RemoveContainer" containerID="8999b99690afa341da3af5fd54a9f5e62e8434ffea0626c95c62873c714d514e"
Nov 01 14:45:04.890616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.890464    5423 scope.go:122] "RemoveContainer" containerID="33673c3c974cd82f25faf1b11260f0c07d7f345f6ff1a9dea7025c3bcb0c20bc"
Nov 01 14:45:04.890852 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:04.890805    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4_odh-ai-gateway-infra(2955b737-485f-42e8-beb3-cda54d0d64a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8"
Nov 01 14:45:04.891897 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.891873    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt" event={"ID":"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9","Type":"ContainerStarted","Data":"b5f07778811abb6c4411176acbe9dac20e0757b525df1976c6f24e778309bb50"}
Nov 01 14:45:04.894548 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.894522    5423 generic.go:356] "Generic (PLEG): container finished" podID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerID="b9dd71be2e50c8b1dd95d0221e13c57f7ca5e99016a1fd643f66de29c2e25891" exitCode=60
Nov 01 14:45:04.894644 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.894599    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" event={"ID":"b8225f9f-a744-4fd9-9678-9b85091004b5","Type":"ContainerDied","Data":"b9dd71be2e50c8b1dd95d0221e13c57f7ca5e99016a1fd643f66de29c2e25891"}
Nov 01 14:45:04.894857 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.894841    5423 scope.go:122] "RemoveContainer" containerID="b9dd71be2e50c8b1dd95d0221e13c57f7ca5e99016a1fd643f66de29c2e25891"
Nov 01 14:45:04.895171 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:04.895149    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp_odh-ai-gateway-infra(b8225f9f-a744-4fd9-9678-9b85091004b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5"
Nov 01 14:45:04.896535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.896515    5423 generic.go:356] "Generic (PLEG): container finished" podID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerID="7d2d51806a6f87ab56c15390de3de122964a782412f66204b4912ce11efb2bd4" exitCode=60
Nov 01 14:45:04.896613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.896541    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" event={"ID":"1b5a0187-9e1e-4b8c-9caa-e505814159ab","Type":"ContainerDied","Data":"7d2d51806a6f87ab56c15390de3de122964a782412f66204b4912ce11efb2bd4"}
Nov 01 14:45:04.896800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.896786    5423 scope.go:122] "RemoveContainer" containerID="7d2d51806a6f87ab56c15390de3de122964a782412f66204b4912ce11efb2bd4"
Nov 01 14:45:04.897060 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:04.897042    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw_odh-ai-gateway-infra(1b5a0187-9e1e-4b8c-9caa-e505814159ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab"
Nov 01 14:45:04.964982 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.964958    5423 scope.go:122] "RemoveContainer" containerID="927b72b710f849ae0a086090e0a6e90fcf48d6cf414f473f32a5ac84e791460d"
Nov 01 14:45:04.978835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:04.978814    5423 scope.go:122] "RemoveContainer" containerID="f09b335b2496dd5ed6c2afb09c85e27190ef2c556975899f5ce3632d72648497"
Nov 01 14:45:05.150850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.150784    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:45:05.152938 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:05.152908    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:45:05.312577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.312552    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab67ded-c1af-46c3-8530-e426b088c1ec-kube-api-access-g6g2j\" (UniqueName: \"kubernetes.io/projected/fab67ded-c1af-46c3-8530-e426b088c1ec-kube-api-access-g6g2j\") pod \"fab67ded-c1af-46c3-8530-e426b088c1ec\" (UID: \"fab67ded-c1af-46c3-8530-e426b088c1ec\") "
Nov 01 14:45:05.314645 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.314622    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab67ded-c1af-46c3-8530-e426b088c1ec-kube-api-access-g6g2j" pod "fab67ded-c1af-46c3-8530-e426b088c1ec" (UID: "fab67ded-c1af-46c3-8530-e426b088c1ec"). InnerVolumeSpecName "kube-api-access-g6g2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:05.414155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.414083    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6g2j\" (UniqueName: \"kubernetes.io/projected/fab67ded-c1af-46c3-8530-e426b088c1ec-kube-api-access-g6g2j\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:05.905592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.905565    5423 scope.go:122] "RemoveContainer" containerID="b9dd71be2e50c8b1dd95d0221e13c57f7ca5e99016a1fd643f66de29c2e25891"
Nov 01 14:45:05.906062 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:05.905935    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp_odh-ai-gateway-infra(b8225f9f-a744-4fd9-9678-9b85091004b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5"
Nov 01 14:45:05.907362 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.907339    5423 scope.go:122] "RemoveContainer" containerID="7d2d51806a6f87ab56c15390de3de122964a782412f66204b4912ce11efb2bd4"
Nov 01 14:45:05.907670 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:05.907644    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw_odh-ai-gateway-infra(1b5a0187-9e1e-4b8c-9caa-e505814159ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab"
Nov 01 14:45:05.908911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.908894    5423 scope.go:122] "RemoveContainer" containerID="33673c3c974cd82f25faf1b11260f0c07d7f345f6ff1a9dea7025c3bcb0c20bc"
Nov 01 14:45:05.909097 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:05.909076    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4_odh-ai-gateway-infra(2955b737-485f-42e8-beb3-cda54d0d64a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8"
Nov 01 14:45:05.910327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.910301    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt" event={"ID":"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9","Type":"ContainerStarted","Data":"835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a"}
Nov 01 14:45:05.910468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.910441    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:05.911715 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.911691    5423 generic.go:356] "Generic (PLEG): container finished" podID="fab67ded-c1af-46c3-8530-e426b088c1ec" containerID="d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9" exitCode=137
Nov 01 14:45:05.911820 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.911720    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-1f0b13"
Nov 01 14:45:05.911820 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.911737    5423 scope.go:122] "RemoveContainer" containerID="d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9"
Nov 01 14:45:05.921437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.921418    5423 scope.go:122] "RemoveContainer" containerID="d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9"
Nov 01 14:45:05.921702 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:05.921680    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9\": container with ID starting with d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9 not found: ID does not exist" containerID="d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9"
Nov 01 14:45:05.921766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.921710    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9"} err="failed to get container status \"d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9\": rpc error: code = NotFound desc = could not find container \"d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9\": container with ID starting with d276ee3a2d135b957275758535a222f02ccc251d870a2a5ff021e36db8afd1d9 not found: ID does not exist"
Nov 01 14:45:05.962204 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:05.962182    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab67ded-c1af-46c3-8530-e426b088c1ec" path="/var/lib/kubelet/pods/fab67ded-c1af-46c3-8530-e426b088c1ec/volumes"
Nov 01 14:45:06.141632 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:06.141574    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt" podStartSLOduration=3.170505518 podStartE2EDuration="4.141557669s" totalImagesPullingTime="971.052151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:06.138850585 +0000 UTC m=+1646.304335883" watchObservedRunningTime="2026-11-01 14:45:06.141557669 +0000 UTC m=+1646.307042964"
Nov 01 14:45:06.777132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:06.777105    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:45:06.894025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:06.893991    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"]
Nov 01 14:45:06.894244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:06.894218    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77" podUID="800238c0-bf47-460e-b0ca-18fba0a9a120" containerName="maas-api" containerID="cri-o://75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a" gracePeriod=30
Nov 01 14:45:07.052792 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.052684    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"]
Nov 01 14:45:07.247360 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.246173    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:45:07.407430 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.407001    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:07.427614 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.427585    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2955b737-485f-42e8-beb3-cda54d0d64a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955b737-485f-42e8-beb3-cda54d0d64a8-maas-api-service-ca\") pod \"2955b737-485f-42e8-beb3-cda54d0d64a8\" (UID: \"2955b737-485f-42e8-beb3-cda54d0d64a8\") "
Nov 01 14:45:07.427761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.427715    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-metrics-certs\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-metrics-certs\") pod \"800238c0-bf47-460e-b0ca-18fba0a9a120\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") "
Nov 01 14:45:07.427761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.427742    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-maas-api-tls\") pod \"800238c0-bf47-460e-b0ca-18fba0a9a120\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") "
Nov 01 14:45:07.427873 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.427848    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800238c0-bf47-460e-b0ca-18fba0a9a120-kube-api-access-gtpvw\" (UniqueName: \"kubernetes.io/projected/800238c0-bf47-460e-b0ca-18fba0a9a120-kube-api-access-gtpvw\") pod \"800238c0-bf47-460e-b0ca-18fba0a9a120\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") "
Nov 01 14:45:07.427999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.427922    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800238c0-bf47-460e-b0ca-18fba0a9a120-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800238c0-bf47-460e-b0ca-18fba0a9a120-openshift-service-ca\") pod \"800238c0-bf47-460e-b0ca-18fba0a9a120\" (UID: \"800238c0-bf47-460e-b0ca-18fba0a9a120\") "
Nov 01 14:45:07.427999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.427953    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2955b737-485f-42e8-beb3-cda54d0d64a8-kube-api-access-5dntv\" (UniqueName: \"kubernetes.io/projected/2955b737-485f-42e8-beb3-cda54d0d64a8-kube-api-access-5dntv\") pod \"2955b737-485f-42e8-beb3-cda54d0d64a8\" (UID: \"2955b737-485f-42e8-beb3-cda54d0d64a8\") "
Nov 01 14:45:07.428131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.427993    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2955b737-485f-42e8-beb3-cda54d0d64a8-maas-api-service-ca" pod "2955b737-485f-42e8-beb3-cda54d0d64a8" (UID: "2955b737-485f-42e8-beb3-cda54d0d64a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:07.428290 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.428266    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800238c0-bf47-460e-b0ca-18fba0a9a120-openshift-service-ca" pod "800238c0-bf47-460e-b0ca-18fba0a9a120" (UID: "800238c0-bf47-460e-b0ca-18fba0a9a120"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:07.428514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.428495    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955b737-485f-42e8-beb3-cda54d0d64a8-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:07.428605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.428520    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/800238c0-bf47-460e-b0ca-18fba0a9a120-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:07.430345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.430317    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-maas-api-tls" pod "800238c0-bf47-460e-b0ca-18fba0a9a120" (UID: "800238c0-bf47-460e-b0ca-18fba0a9a120"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:45:07.430618 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.430583    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800238c0-bf47-460e-b0ca-18fba0a9a120-kube-api-access-gtpvw" pod "800238c0-bf47-460e-b0ca-18fba0a9a120" (UID: "800238c0-bf47-460e-b0ca-18fba0a9a120"). InnerVolumeSpecName "kube-api-access-gtpvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:07.430704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.430677    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-metrics-certs" pod "800238c0-bf47-460e-b0ca-18fba0a9a120" (UID: "800238c0-bf47-460e-b0ca-18fba0a9a120"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:45:07.430821 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.430793    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2955b737-485f-42e8-beb3-cda54d0d64a8-kube-api-access-5dntv" pod "2955b737-485f-42e8-beb3-cda54d0d64a8" (UID: "2955b737-485f-42e8-beb3-cda54d0d64a8"). InnerVolumeSpecName "kube-api-access-5dntv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:07.530330 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.530291    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:07.530330 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.530331    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/800238c0-bf47-460e-b0ca-18fba0a9a120-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:07.530544 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.530348    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtpvw\" (UniqueName: \"kubernetes.io/projected/800238c0-bf47-460e-b0ca-18fba0a9a120-kube-api-access-gtpvw\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:07.530544 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.530365    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dntv\" (UniqueName: \"kubernetes.io/projected/2955b737-485f-42e8-beb3-cda54d0d64a8-kube-api-access-5dntv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:07.943925 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.943895    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"
Nov 01 14:45:07.944098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.943898    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4" event={"ID":"2955b737-485f-42e8-beb3-cda54d0d64a8","Type":"ContainerDied","Data":"47c170d9d80d2deeea2258a3288711ba22c7ad93010b7593ad36d59111802969"}
Nov 01 14:45:07.944098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.944040    5423 scope.go:122] "RemoveContainer" containerID="33673c3c974cd82f25faf1b11260f0c07d7f345f6ff1a9dea7025c3bcb0c20bc"
Nov 01 14:45:07.945461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.945277    5423 generic.go:356] "Generic (PLEG): container finished" podID="800238c0-bf47-460e-b0ca-18fba0a9a120" containerID="75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a" exitCode=0
Nov 01 14:45:07.945461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.945316    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"
Nov 01 14:45:07.945461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.945338    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77" event={"ID":"800238c0-bf47-460e-b0ca-18fba0a9a120","Type":"ContainerDied","Data":"75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a"}
Nov 01 14:45:07.945461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.945368    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77" event={"ID":"800238c0-bf47-460e-b0ca-18fba0a9a120","Type":"ContainerDied","Data":"bda070f752cc8a29e5aad2c0e835a4bcc0c86b53bc68e293da5adb9c55a06f3d"}
Nov 01 14:45:07.955166 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.955136    5423 scope.go:122] "RemoveContainer" containerID="75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a"
Nov 01 14:45:07.965369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.965352    5423 scope.go:122] "RemoveContainer" containerID="75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a"
Nov 01 14:45:07.965661 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:07.965640    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a\": container with ID starting with 75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a not found: ID does not exist" containerID="75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a"
Nov 01 14:45:07.965729 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:07.965668    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a"} err="failed to get container status \"75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a\": rpc error: code = NotFound desc = could not find container \"75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a\": container with ID starting with 75de919cfb0c8d918e4d2f732590983bb2a0d760df77cb4837b9e55e353f482a not found: ID does not exist"
Nov 01 14:45:08.111691 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:08.111656    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"]
Nov 01 14:45:08.114726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:08.114695    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4976-76f69455f8-v5b77"]
Nov 01 14:45:08.224842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:08.224751    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"]
Nov 01 14:45:08.229425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:08.229395    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0c4976-29892405-sr7j4"]
Nov 01 14:45:09.964749 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:09.964717    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8" path="/var/lib/kubelet/pods/2955b737-485f-42e8-beb3-cda54d0d64a8/volumes"
Nov 01 14:45:09.965193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:09.965179    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800238c0-bf47-460e-b0ca-18fba0a9a120" path="/var/lib/kubelet/pods/800238c0-bf47-460e-b0ca-18fba0a9a120/volumes"
Nov 01 14:45:11.940256 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:11.940229    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:45:14.006705 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:14.006674    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"]
Nov 01 14:45:14.007131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:14.006870    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy" containerID="cri-o://fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876" gracePeriod=30
Nov 01 14:45:14.953485 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:14.953455    5423 scope.go:122] "RemoveContainer" containerID="b9dd71be2e50c8b1dd95d0221e13c57f7ca5e99016a1fd643f66de29c2e25891"
Nov 01 14:45:16.001717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:16.001685    5423 generic.go:356] "Generic (PLEG): container finished" podID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerID="207a11e7cbce2d96c135419003cca92f0c2e4e74d93d86f4157ab430eb3740d7" exitCode=60
Nov 01 14:45:16.002135 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:16.001760    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" event={"ID":"b8225f9f-a744-4fd9-9678-9b85091004b5","Type":"ContainerDied","Data":"207a11e7cbce2d96c135419003cca92f0c2e4e74d93d86f4157ab430eb3740d7"}
Nov 01 14:45:16.002135 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:16.001828    5423 scope.go:122] "RemoveContainer" containerID="b9dd71be2e50c8b1dd95d0221e13c57f7ca5e99016a1fd643f66de29c2e25891"
Nov 01 14:45:16.002251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:16.002236    5423 scope.go:122] "RemoveContainer" containerID="207a11e7cbce2d96c135419003cca92f0c2e4e74d93d86f4157ab430eb3740d7"
Nov 01 14:45:16.002619 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:16.002596    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp_odh-ai-gateway-infra(b8225f9f-a744-4fd9-9678-9b85091004b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5"
Nov 01 14:45:16.953565 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:16.953532    5423 scope.go:122] "RemoveContainer" containerID="7d2d51806a6f87ab56c15390de3de122964a782412f66204b4912ce11efb2bd4"
Nov 01 14:45:17.137147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:17.137114    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"]
Nov 01 14:45:17.293198 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:17.293176    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:17.356725 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:17.356696    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8225f9f-a744-4fd9-9678-9b85091004b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8225f9f-a744-4fd9-9678-9b85091004b5-maas-api-service-ca\") pod \"b8225f9f-a744-4fd9-9678-9b85091004b5\" (UID: \"b8225f9f-a744-4fd9-9678-9b85091004b5\") "
Nov 01 14:45:17.356936 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:17.356923    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8225f9f-a744-4fd9-9678-9b85091004b5-kube-api-access-l8rf2\" (UniqueName: \"kubernetes.io/projected/b8225f9f-a744-4fd9-9678-9b85091004b5-kube-api-access-l8rf2\") pod \"b8225f9f-a744-4fd9-9678-9b85091004b5\" (UID: \"b8225f9f-a744-4fd9-9678-9b85091004b5\") "
Nov 01 14:45:17.357079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:17.357057    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8225f9f-a744-4fd9-9678-9b85091004b5-maas-api-service-ca" pod "b8225f9f-a744-4fd9-9678-9b85091004b5" (UID: "b8225f9f-a744-4fd9-9678-9b85091004b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:17.357288 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:17.357275    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8225f9f-a744-4fd9-9678-9b85091004b5-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:17.359020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:17.359001    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8225f9f-a744-4fd9-9678-9b85091004b5-kube-api-access-l8rf2" pod "b8225f9f-a744-4fd9-9678-9b85091004b5" (UID: "b8225f9f-a744-4fd9-9678-9b85091004b5"). InnerVolumeSpecName "kube-api-access-l8rf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:17.458861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:17.458833    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8rf2\" (UniqueName: \"kubernetes.io/projected/b8225f9f-a744-4fd9-9678-9b85091004b5-kube-api-access-l8rf2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:18.016369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.016332    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp" event={"ID":"b8225f9f-a744-4fd9-9678-9b85091004b5","Type":"ContainerDied","Data":"e2d42013fec854b1c93fa02bedc0c76a53b92673bcf951137a6603a5fe4194d9"}
Nov 01 14:45:18.016533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.016349    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"
Nov 01 14:45:18.016533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.016433    5423 scope.go:122] "RemoveContainer" containerID="207a11e7cbce2d96c135419003cca92f0c2e4e74d93d86f4157ab430eb3740d7"
Nov 01 14:45:18.017277 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.017247    5423 patch_prober.go:29] interesting pod/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:45:18.017425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.017300    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:45:18.018817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.018792    5423 generic.go:356] "Generic (PLEG): container finished" podID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerID="011db98cd3154606be55f1196b0918e1742f741266032a35bd00c52f71512be4" exitCode=60
Nov 01 14:45:18.018979 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.018944    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" event={"ID":"1b5a0187-9e1e-4b8c-9caa-e505814159ab","Type":"ContainerDied","Data":"011db98cd3154606be55f1196b0918e1742f741266032a35bd00c52f71512be4"}
Nov 01 14:45:18.019362 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.019347    5423 scope.go:122] "RemoveContainer" containerID="011db98cd3154606be55f1196b0918e1742f741266032a35bd00c52f71512be4"
Nov 01 14:45:18.019621 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:18.019605    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw_odh-ai-gateway-infra(1b5a0187-9e1e-4b8c-9caa-e505814159ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab"
Nov 01 14:45:18.026207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.026185    5423 scope.go:122] "RemoveContainer" containerID="7d2d51806a6f87ab56c15390de3de122964a782412f66204b4912ce11efb2bd4"
Nov 01 14:45:18.191097 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.191065    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"]
Nov 01 14:45:18.193997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:18.193969    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892405-59rmp"]
Nov 01 14:45:19.292896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.292839    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:45:19.322260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.322228    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"]
Nov 01 14:45:19.405052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405025    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-kube-api-access-9ltt9\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-kube-api-access-9ltt9\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405082    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-podinfo\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405115    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-data\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405186    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-certs\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405222    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb43700-7fa7-457a-be62-bb3be96e9b10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb43700-7fa7-457a-be62-bb3be96e9b10-istiod-ca-cert\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405274    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-envoy\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405563 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405542    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-token\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-token\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405585    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-socket\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405670 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405643    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-credential-socket\") pod \"0bb43700-7fa7-457a-be62-bb3be96e9b10\" (UID: \"0bb43700-7fa7-457a-be62-bb3be96e9b10\") "
Nov 01 14:45:19.405824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.405801    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-data" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:19.406138 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.406116    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-socket" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:19.406238 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.406222    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.406296 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.406240    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.406363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.406334    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-credential-socket" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:19.406637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.406582    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb43700-7fa7-457a-be62-bb3be96e9b10-istiod-ca-cert" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:19.407045 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.407024    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-certs" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:19.409021 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.408997    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-kube-api-access-9ltt9" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "kube-api-access-9ltt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:19.409231 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.409199    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-token" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:19.411904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.411870    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-envoy" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:19.412319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.412292    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-podinfo" pod "0bb43700-7fa7-457a-be62-bb3be96e9b10" (UID: "0bb43700-7fa7-457a-be62-bb3be96e9b10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:45:19.474656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.474626    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:19.507516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.507486    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.507516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.507513    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ltt9\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-kube-api-access-9ltt9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.507516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.507523    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.507749 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.507533    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.507749 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.507542    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb43700-7fa7-457a-be62-bb3be96e9b10-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.507749 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.507550    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.507749 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.507581    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb43700-7fa7-457a-be62-bb3be96e9b10-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.608143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.608111    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5a0187-9e1e-4b8c-9caa-e505814159ab-kube-api-access-4kvh7\" (UniqueName: \"kubernetes.io/projected/1b5a0187-9e1e-4b8c-9caa-e505814159ab-kube-api-access-4kvh7\") pod \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\" (UID: \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\") "
Nov 01 14:45:19.608322 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.608231    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5a0187-9e1e-4b8c-9caa-e505814159ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5a0187-9e1e-4b8c-9caa-e505814159ab-maas-api-service-ca\") pod \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\" (UID: \"1b5a0187-9e1e-4b8c-9caa-e505814159ab\") "
Nov 01 14:45:19.608652 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.608621    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5a0187-9e1e-4b8c-9caa-e505814159ab-maas-api-service-ca" pod "1b5a0187-9e1e-4b8c-9caa-e505814159ab" (UID: "1b5a0187-9e1e-4b8c-9caa-e505814159ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:19.608880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.608863    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5a0187-9e1e-4b8c-9caa-e505814159ab-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.610443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.610421    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5a0187-9e1e-4b8c-9caa-e505814159ab-kube-api-access-4kvh7" pod "1b5a0187-9e1e-4b8c-9caa-e505814159ab" (UID: "1b5a0187-9e1e-4b8c-9caa-e505814159ab"). InnerVolumeSpecName "kube-api-access-4kvh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:19.712408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.711708    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kvh7\" (UniqueName: \"kubernetes.io/projected/1b5a0187-9e1e-4b8c-9caa-e505814159ab-kube-api-access-4kvh7\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:19.960671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:19.960599    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" path="/var/lib/kubelet/pods/b8225f9f-a744-4fd9-9678-9b85091004b5/volumes"
Nov 01 14:45:20.092587 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.092560    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"]
Nov 01 14:45:20.093602 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.093585    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"
Nov 01 14:45:20.093708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.093630    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw" event={"ID":"1b5a0187-9e1e-4b8c-9caa-e505814159ab","Type":"ContainerDied","Data":"8cd5c532ae1e676446f0cee8c24f09f023f1d8c22d44b5cc22ae462ee6f202d6"}
Nov 01 14:45:20.093708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.093691    5423 scope.go:122] "RemoveContainer" containerID="011db98cd3154606be55f1196b0918e1742f741266032a35bd00c52f71512be4"
Nov 01 14:45:20.095922 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.095901    5423 generic.go:356] "Generic (PLEG): container finished" podID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerID="fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876" exitCode=0
Nov 01 14:45:20.096004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.095963    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"
Nov 01 14:45:20.096064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.095959    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" event={"ID":"0bb43700-7fa7-457a-be62-bb3be96e9b10","Type":"ContainerDied","Data":"fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876"}
Nov 01 14:45:20.096118 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.096099    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv" event={"ID":"0bb43700-7fa7-457a-be62-bb3be96e9b10","Type":"ContainerDied","Data":"548bf7775bf170556f4ffbcee36ca0ab79cab0d4aacc6ad9be490982ba27ee13"}
Nov 01 14:45:20.104113 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.104099    5423 scope.go:122] "RemoveContainer" containerID="fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876"
Nov 01 14:45:20.112783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.112768    5423 scope.go:122] "RemoveContainer" containerID="12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea"
Nov 01 14:45:20.124045 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.124019    5423 scope.go:122] "RemoveContainer" containerID="fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876"
Nov 01 14:45:20.124479 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:20.124446    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876\": container with ID starting with fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876 not found: ID does not exist" containerID="fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876"
Nov 01 14:45:20.124609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.124478    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876"} err="failed to get container status \"fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876\": rpc error: code = NotFound desc = could not find container \"fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876\": container with ID starting with fe1b50263e73b4a7eaff327ef0af7c976594aef4ef6c023a1ed970de5473a876 not found: ID does not exist"
Nov 01 14:45:20.124609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.124597    5423 scope.go:122] "RemoveContainer" containerID="12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea"
Nov 01 14:45:20.125102 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:20.125081    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea\": container with ID starting with 12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea not found: ID does not exist" containerID="12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea"
Nov 01 14:45:20.125191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.125108    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea"} err="failed to get container status \"12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea\": rpc error: code = NotFound desc = could not find container \"12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea\": container with ID starting with 12291758cb0243aeed0c9b987473772fbee76aade461c4eaafe60c5a699924ea not found: ID does not exist"
Nov 01 14:45:20.188462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.188432    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"]
Nov 01 14:45:20.324731 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.324702    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"]
Nov 01 14:45:20.330925 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.330901    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26a0c1-29892405-qcllw"]
Nov 01 14:45:20.448230 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.448194    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"]
Nov 01 14:45:20.460053 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:20.460028    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0c4976-openshift-default-59c8cf666c-dscsv"]
Nov 01 14:45:21.960531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:21.960502    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" path="/var/lib/kubelet/pods/0bb43700-7fa7-457a-be62-bb3be96e9b10/volumes"
Nov 01 14:45:21.960937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:21.960923    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" path="/var/lib/kubelet/pods/1b5a0187-9e1e-4b8c-9caa-e505814159ab/volumes"
Nov 01 14:45:25.093690 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.093631    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb" podUID="51384a52-f8b5-414b-b184-60992afb1977" containerName="payload-processing" containerID="cri-o://90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133" gracePeriod=25
Nov 01 14:45:25.189022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.188986    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5" podUID="4d6aff9c-21fa-4506-8187-e172cdd0b6a3" containerName="payload-pre-processing" containerID="cri-o://a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74" gracePeriod=25
Nov 01 14:45:25.468164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.468138    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:45:25.499010 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.498989    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:45:25.502912 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.502892    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51384a52-f8b5-414b-b184-60992afb1977-kube-api-access-5xn4q\" (UniqueName: \"kubernetes.io/projected/51384a52-f8b5-414b-b184-60992afb1977-kube-api-access-5xn4q\") pod \"51384a52-f8b5-414b-b184-60992afb1977\" (UID: \"51384a52-f8b5-414b-b184-60992afb1977\") "
Nov 01 14:45:25.503131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.503113    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51384a52-f8b5-414b-b184-60992afb1977-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51384a52-f8b5-414b-b184-60992afb1977-plugins-config-volume\") pod \"51384a52-f8b5-414b-b184-60992afb1977\" (UID: \"51384a52-f8b5-414b-b184-60992afb1977\") "
Nov 01 14:45:25.503500 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.503477    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51384a52-f8b5-414b-b184-60992afb1977-plugins-config-volume" pod "51384a52-f8b5-414b-b184-60992afb1977" (UID: "51384a52-f8b5-414b-b184-60992afb1977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:25.504957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.504934    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51384a52-f8b5-414b-b184-60992afb1977-kube-api-access-5xn4q" pod "51384a52-f8b5-414b-b184-60992afb1977" (UID: "51384a52-f8b5-414b-b184-60992afb1977"). InnerVolumeSpecName "kube-api-access-5xn4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:25.604070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.604039    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-kube-api-access-z5zrl\" (UniqueName: \"kubernetes.io/projected/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-kube-api-access-z5zrl\") pod \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\" (UID: \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\") "
Nov 01 14:45:25.604232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.604098    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-plugins-config-volume\") pod \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\" (UID: \"4d6aff9c-21fa-4506-8187-e172cdd0b6a3\") "
Nov 01 14:45:25.604547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.604521    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-plugins-config-volume" pod "4d6aff9c-21fa-4506-8187-e172cdd0b6a3" (UID: "4d6aff9c-21fa-4506-8187-e172cdd0b6a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:25.604653 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.604637    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xn4q\" (UniqueName: \"kubernetes.io/projected/51384a52-f8b5-414b-b184-60992afb1977-kube-api-access-5xn4q\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:25.604732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.604655    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:25.604732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.604665    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51384a52-f8b5-414b-b184-60992afb1977-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:25.606117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.606059    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-kube-api-access-z5zrl" pod "4d6aff9c-21fa-4506-8187-e172cdd0b6a3" (UID: "4d6aff9c-21fa-4506-8187-e172cdd0b6a3"). InnerVolumeSpecName "kube-api-access-z5zrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:25.706304 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:25.706276    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5zrl\" (UniqueName: \"kubernetes.io/projected/4d6aff9c-21fa-4506-8187-e172cdd0b6a3-kube-api-access-z5zrl\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:26.136018 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.135988    5423 generic.go:356] "Generic (PLEG): container finished" podID="51384a52-f8b5-414b-b184-60992afb1977" containerID="90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133" exitCode=0
Nov 01 14:45:26.136478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.136033    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"
Nov 01 14:45:26.136478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.136071    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb" event={"ID":"51384a52-f8b5-414b-b184-60992afb1977","Type":"ContainerDied","Data":"90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133"}
Nov 01 14:45:26.136478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.136132    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb" event={"ID":"51384a52-f8b5-414b-b184-60992afb1977","Type":"ContainerDied","Data":"86e577ac6da09d2eb45e7bfdb4918cd8e9d355e63e49cd902feb7c6d0063db8e"}
Nov 01 14:45:26.136478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.136164    5423 scope.go:122] "RemoveContainer" containerID="90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133"
Nov 01 14:45:26.137560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.137541    5423 generic.go:356] "Generic (PLEG): container finished" podID="4d6aff9c-21fa-4506-8187-e172cdd0b6a3" containerID="a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74" exitCode=0
Nov 01 14:45:26.137642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.137578    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"
Nov 01 14:45:26.137642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.137589    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5" event={"ID":"4d6aff9c-21fa-4506-8187-e172cdd0b6a3","Type":"ContainerDied","Data":"a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74"}
Nov 01 14:45:26.137744 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.137643    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5" event={"ID":"4d6aff9c-21fa-4506-8187-e172cdd0b6a3","Type":"ContainerDied","Data":"f12de89a3d797e6ef34e5f5e40237ef9f568a38d9d467b55be4a847aa0420c3e"}
Nov 01 14:45:26.145846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.145829    5423 scope.go:122] "RemoveContainer" containerID="90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133"
Nov 01 14:45:26.146110 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:26.146092    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133\": container with ID starting with 90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133 not found: ID does not exist" containerID="90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133"
Nov 01 14:45:26.146155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.146117    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133"} err="failed to get container status \"90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133\": rpc error: code = NotFound desc = could not find container \"90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133\": container with ID starting with 90ddf2a928294968e8cedbdac986cc87bc4d2771ae4fe8ce29056d6ca22b4133 not found: ID does not exist"
Nov 01 14:45:26.146155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.146133    5423 scope.go:122] "RemoveContainer" containerID="a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74"
Nov 01 14:45:26.154595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.154575    5423 scope.go:122] "RemoveContainer" containerID="a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74"
Nov 01 14:45:26.154857 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:26.154836    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74\": container with ID starting with a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74 not found: ID does not exist" containerID="a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74"
Nov 01 14:45:26.154920 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.154868    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74"} err="failed to get container status \"a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74\": rpc error: code = NotFound desc = could not find container \"a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74\": container with ID starting with a2bbc34755f70a5681138b661e61f0177f1a77b7262d4924d902736225287e74 not found: ID does not exist"
Nov 01 14:45:26.305067 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.304980    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"]
Nov 01 14:45:26.356087 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.356049    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26a0c1-6b847cd97f-8r9xb"]
Nov 01 14:45:26.417915 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.417878    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"]
Nov 01 14:45:26.421264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:26.421241    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26a0c1-688f9467b7-vk4q5"]
Nov 01 14:45:27.960992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:27.960951    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6aff9c-21fa-4506-8187-e172cdd0b6a3" path="/var/lib/kubelet/pods/4d6aff9c-21fa-4506-8187-e172cdd0b6a3/volumes"
Nov 01 14:45:27.961576 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:27.961556    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51384a52-f8b5-414b-b184-60992afb1977" path="/var/lib/kubelet/pods/51384a52-f8b5-414b-b184-60992afb1977/volumes"
Nov 01 14:45:35.149563 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:45:35.149531    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:45:43.504544 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:45:43.504506    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:46:02.585205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.585169    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"]
Nov 01 14:46:02.585952 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.585935    5423 cpu_manager.go:414] "containerMap: removing container" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8" containerName="cleanup"
Nov 01 14:46:02.585999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.585954    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8" containerName="cleanup"
Nov 01 14:46:02.585999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.585970    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy"
Nov 01 14:46:02.585999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.585978    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy"
Nov 01 14:46:02.585999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.585995    5423 cpu_manager.go:414] "containerMap: removing container" podUID="51384a52-f8b5-414b-b184-60992afb1977" containerName="payload-processing"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586004    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51384a52-f8b5-414b-b184-60992afb1977" containerName="payload-processing"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586016    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586023    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586045    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586053    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586076    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4d6aff9c-21fa-4506-8187-e172cdd0b6a3" containerName="payload-pre-processing"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586083    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6aff9c-21fa-4506-8187-e172cdd0b6a3" containerName="payload-pre-processing"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586104    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:02.586136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586111    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586140    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586147    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586160    5423 cpu_manager.go:414] "containerMap: removing container" podUID="800238c0-bf47-460e-b0ca-18fba0a9a120" containerName="maas-api"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586167    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800238c0-bf47-460e-b0ca-18fba0a9a120" containerName="maas-api"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586268    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586278    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586289    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586299    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586309    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="800238c0-bf47-460e-b0ca-18fba0a9a120" containerName="maas-api"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586318    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586329    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586338    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="51384a52-f8b5-414b-b184-60992afb1977" containerName="payload-processing"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586348    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586357    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586369    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy"
Nov 01 14:46:02.586475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.586379    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6aff9c-21fa-4506-8187-e172cdd0b6a3" containerName="payload-pre-processing"
Nov 01 14:46:02.590138 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.590116    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.644209 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.644177    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-be7e2e-openshift-default-dockercfg-jnbbc\""
Nov 01 14:46:02.648806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.648778    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-certs\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.648933 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.648849    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-socket\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.649013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.648987    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbts\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-kube-api-access-2cbts\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.649069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.649052    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-envoy\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.649519 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.649100    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-podinfo\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.649519 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.649199    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-data\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.649519 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.649248    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-credential-socket\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.649519 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.649372    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istiod-ca-cert\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.649519 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.649434    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-token\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.697725 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.697691    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"]
Nov 01 14:46:02.750480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750446    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-podinfo\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.750659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750489    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-data\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.750659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750513    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-credential-socket\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.750659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750641    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istiod-ca-cert\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.750863 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750728    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-token\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.750863 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750827    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-certs\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.750988 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750911    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-socket\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.750988 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750935    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-data\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.750988 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.750966    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbts\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-kube-api-access-2cbts\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.751142 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.751007    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-credential-socket\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.751142 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.751017    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-envoy\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.751242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.751215    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-certs\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.751486 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.751470    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-socket\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.751938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.751917    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istiod-ca-cert\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.753288 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.753260    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-envoy\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.753702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.753680    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-podinfo\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.810769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.810739    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-token\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.811459 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.811436    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbts\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-kube-api-access-2cbts\") pod \"e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") " pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:02.904997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:02.904911    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:03.478599 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:03.478554    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b08385e_e72e_4078_9dce_f385d0ba5ad3.slice/crio-711d6b9f81a9806cb468ae252f1c81ade89919a6e2155e4e57d3ff3ff3ba8171 WatchSource:0}: Error finding container 711d6b9f81a9806cb468ae252f1c81ade89919a6e2155e4e57d3ff3ff3ba8171: Status 404 returned error can't find the container with id 711d6b9f81a9806cb468ae252f1c81ade89919a6e2155e4e57d3ff3ff3ba8171
Nov 01 14:46:03.481042 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:03.481008    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:46:03.481155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:03.481090    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:46:03.481155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:03.481134    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:46:03.481498 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:03.481472    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"]
Nov 01 14:46:04.427307 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:04.427266    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" event={"ID":"0b08385e-e72e-4078-9dce-f385d0ba5ad3","Type":"ContainerStarted","Data":"baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8"}
Nov 01 14:46:04.427856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:04.427369    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" event={"ID":"0b08385e-e72e-4078-9dce-f385d0ba5ad3","Type":"ContainerStarted","Data":"711d6b9f81a9806cb468ae252f1c81ade89919a6e2155e4e57d3ff3ff3ba8171"}
Nov 01 14:46:04.608475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:04.608421    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" podStartSLOduration=2.608404772 podStartE2EDuration="2.608404772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:04.605566795 +0000 UTC m=+1704.771052093" watchObservedRunningTime="2026-11-01 14:46:04.608404772 +0000 UTC m=+1704.773890065"
Nov 01 14:46:04.905030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:04.904993    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:04.911189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:04.911163    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:05.149789 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:46:05.149754    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:46:05.433631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:05.433600    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:05.434693 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:05.434669    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:46:11.614276 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614246    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"]
Nov 01 14:46:11.614813 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614798    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:11.614856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614813    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:11.614856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614841    5423 cpu_manager.go:414] "containerMap: removing container" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8" containerName="cleanup"
Nov 01 14:46:11.614856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614846    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2955b737-485f-42e8-beb3-cda54d0d64a8" containerName="cleanup"
Nov 01 14:46:11.614856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614854    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy"
Nov 01 14:46:11.614990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614858    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb43700-7fa7-457a-be62-bb3be96e9b10" containerName="istio-proxy"
Nov 01 14:46:11.614990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614875    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:11.614990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614880    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8225f9f-a744-4fd9-9678-9b85091004b5" containerName="cleanup"
Nov 01 14:46:11.615077 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.614994    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5a0187-9e1e-4b8c-9caa-e505814159ab" containerName="cleanup"
Nov 01 14:46:11.618605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.618587    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:11.670870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.670839    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"]
Nov 01 14:46:11.680412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.680374    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-be7e2e\""
Nov 01 14:46:11.680613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.680398    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-be7e2e-dockercfg-2rjrz\""
Nov 01 14:46:11.724571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.724542    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"]
Nov 01 14:46:11.728577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.728560    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:11.742582 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.742561    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtz2m\" (UniqueName: \"kubernetes.io/projected/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-kube-api-access-mtz2m\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk\" (UID: \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:11.742741 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.742726    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk\" (UID: \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:11.783108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.783074    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"]
Nov 01 14:46:11.921104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.909354    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk\" (UID: \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:11.921104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.909438    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2\" (UID: \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:11.921104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.909582    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtz2m\" (UniqueName: \"kubernetes.io/projected/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-kube-api-access-mtz2m\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk\" (UID: \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:11.921104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.909647    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmt7l\" (UniqueName: \"kubernetes.io/projected/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-kube-api-access-zmt7l\") pod \"payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2\" (UID: \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:11.921104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.910709    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk\" (UID: \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:11.972424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:11.972361    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtz2m\" (UniqueName: \"kubernetes.io/projected/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-kube-api-access-mtz2m\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk\" (UID: \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:12.010652 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:12.010614    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmt7l\" (UniqueName: \"kubernetes.io/projected/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-kube-api-access-zmt7l\") pod \"payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2\" (UID: \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:12.010828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:12.010767    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2\" (UID: \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:12.011420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:12.011404    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2\" (UID: \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:12.070079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:12.070042    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmt7l\" (UniqueName: \"kubernetes.io/projected/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-kube-api-access-zmt7l\") pod \"payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2\" (UID: \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:12.233743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:12.233671    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:12.342581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:12.342544    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:12.607048 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:12.607018    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ef7c29_67fd_4e8b_b799_8f23ba3cc78a.slice/crio-c1b01a27411902d612a806956912d12ff2fc531bf1fb3186e59eccedf12756d7 WatchSource:0}: Error finding container c1b01a27411902d612a806956912d12ff2fc531bf1fb3186e59eccedf12756d7: Status 404 returned error can't find the container with id c1b01a27411902d612a806956912d12ff2fc531bf1fb3186e59eccedf12756d7
Nov 01 14:46:12.609934 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:12.609912    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"]
Nov 01 14:46:12.716200 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:12.716169    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6ed4d2_ac42_44e3_b3ab_b74ecc8dba34.slice/crio-d3e5817172701b82820d0cbc653c455f71628ed64737f27f83a1425e5ee0370b WatchSource:0}: Error finding container d3e5817172701b82820d0cbc653c455f71628ed64737f27f83a1425e5ee0370b: Status 404 returned error can't find the container with id d3e5817172701b82820d0cbc653c455f71628ed64737f27f83a1425e5ee0370b
Nov 01 14:46:12.719748 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:12.719722    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"]
Nov 01 14:46:13.485972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:13.485934    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2" event={"ID":"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34","Type":"ContainerStarted","Data":"aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979"}
Nov 01 14:46:13.486177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:13.486119    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:13.486177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:13.486152    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2" event={"ID":"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34","Type":"ContainerStarted","Data":"d3e5817172701b82820d0cbc653c455f71628ed64737f27f83a1425e5ee0370b"}
Nov 01 14:46:13.487523 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:13.487495    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk" event={"ID":"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a","Type":"ContainerStarted","Data":"634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d"}
Nov 01 14:46:13.487647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:13.487535    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk" event={"ID":"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a","Type":"ContainerStarted","Data":"c1b01a27411902d612a806956912d12ff2fc531bf1fb3186e59eccedf12756d7"}
Nov 01 14:46:13.487647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:13.487631    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:13.734803 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:13.734750    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk" podStartSLOduration=2.734736116 podStartE2EDuration="2.734736116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:13.73200764 +0000 UTC m=+1713.897492950" watchObservedRunningTime="2026-11-01 14:46:13.734736116 +0000 UTC m=+1713.900221414"
Nov 01 14:46:13.735306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:13.734870    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2" podStartSLOduration=2.734865976 podStartE2EDuration="2.734865976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:13.607693364 +0000 UTC m=+1713.773178663" watchObservedRunningTime="2026-11-01 14:46:13.734865976 +0000 UTC m=+1713.900351307"
Nov 01 14:46:19.030031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:19.029984    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"]
Nov 01 14:46:19.030575 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:19.030225    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" containerID="cri-o://f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce" gracePeriod=30
Nov 01 14:46:19.041508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:19.041484    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"]
Nov 01 14:46:19.041684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:19.041665    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy" containerID="cri-o://1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f" gracePeriod=30
Nov 01 14:46:19.238444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:19.238408    5423 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:46:19.238614 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:19.238470    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:46:20.688724 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.688649    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"]
Nov 01 14:46:20.692942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.692921    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.752692 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.752669    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-34d384-openshift-default-dockercfg-7trgk\""
Nov 01 14:46:20.803424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.803377    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"]
Nov 01 14:46:20.857091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.857038    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.857445 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.857421    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.857630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.857615    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.857797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.857773    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.857930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.857829    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.857930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.857901    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.858050 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.857931    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.858050 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.858032    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45dz\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-kube-api-access-t45dz\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.858156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.858091    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.959606 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959516    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.959606 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959568    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.959606 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959601    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.959895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959654    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.959895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959677    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.959895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959743    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45dz\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-kube-api-access-t45dz\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.959895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959789    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.959895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959864    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.960098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959904    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.960098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.959936    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.960227 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.960196    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.960227 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.960212    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.960344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.960249    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.960587 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.960563    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.962070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.962048    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:20.962164 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:20.962131    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:21.024930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:21.024884    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:21.034884 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:21.034859    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45dz\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-kube-api-access-t45dz\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:21.307183 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:21.307153    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:21.777049 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:21.777015    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2be8d74_4c65_4c2b_a1bb_9df37bc7abd4.slice/crio-6b55c1abcb6c5519e4692016a8571363c69d10a7c32b04a9896b8f4eae02e5e1 WatchSource:0}: Error finding container 6b55c1abcb6c5519e4692016a8571363c69d10a7c32b04a9896b8f4eae02e5e1: Status 404 returned error can't find the container with id 6b55c1abcb6c5519e4692016a8571363c69d10a7c32b04a9896b8f4eae02e5e1
Nov 01 14:46:21.779142 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:21.779116    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"]
Nov 01 14:46:21.779543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:21.779516    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:46:21.779604 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:21.779579    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:46:21.779641 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:21.779628    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:46:22.550885 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:22.550841    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" event={"ID":"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4","Type":"ContainerStarted","Data":"8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972"}
Nov 01 14:46:22.551073 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:22.550903    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" event={"ID":"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4","Type":"ContainerStarted","Data":"6b55c1abcb6c5519e4692016a8571363c69d10a7c32b04a9896b8f4eae02e5e1"}
Nov 01 14:46:22.724566 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:22.724515    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" podStartSLOduration=2.724501291 podStartE2EDuration="2.724501291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:22.721520123 +0000 UTC m=+1722.887005431" watchObservedRunningTime="2026-11-01 14:46:22.724501291 +0000 UTC m=+1722.889986671"
Nov 01 14:46:23.308127 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:23.308095    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:23.313619 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:23.313596    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:23.557818 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:23.557789    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:23.558815 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:23.558773    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:46:24.324818 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.324793    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:46:24.363107 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.363083    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:46:24.401253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401221    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-token\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-token\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.401426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401260    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istiod-ca-cert\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.401426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401288    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-podinfo\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.401426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401334    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-socket\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.401426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401412    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-credential-socket\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.401713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401469    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-podinfo\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.401713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401539    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab4e7d0-7531-4855-b771-f36569ff5400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab4e7d0-7531-4855-b771-f36569ff5400-istiod-ca-cert\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.401713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401583    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-envoy\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.401713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401626    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-certs\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.401713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401686    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-certs\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.401713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401649    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-credential-socket" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401746    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-kube-api-access-cj8n2\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-kube-api-access-cj8n2\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401759    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istiod-ca-cert" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401788    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-data\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401851    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-token\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-token\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401907    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-kube-api-access-jph7t\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-kube-api-access-jph7t\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401956    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-certs" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401962    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-credential-socket\") pod \"7ab4e7d0-7531-4855-b771-f36569ff5400\" (UID: \"7ab4e7d0-7531-4855-b771-f36569ff5400\") "
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401978    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-socket" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.402000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.401984    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab4e7d0-7531-4855-b771-f36569ff5400-istiod-ca-cert" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:46:24.402501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402032    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-data\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.402501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402092    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-socket\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.402501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402160    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-envoy\") pod \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\" (UID: \"a12fcf30-ae85-4ba7-8228-768491bd5ee3\") "
Nov 01 14:46:24.402501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402190    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-credential-socket" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.402501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402200    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-certs" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.402501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402247    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-data" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.402814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402718    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.402814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402740    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab4e7d0-7531-4855-b771-f36569ff5400-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.402814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402756    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.402814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402770    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.402814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402783    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.402814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402796    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.402814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402810    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.403195 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.402823    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.404339 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404049    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-token" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:24.404339 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404331    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-podinfo" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:46:24.404556 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404416    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-token" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:24.404556 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404513    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-podinfo" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:46:24.404665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404554    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-socket" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.404665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404600    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-data" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.404665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404622    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-kube-api-access-cj8n2" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "kube-api-access-cj8n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:24.404854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404792    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-envoy" pod "7ab4e7d0-7531-4855-b771-f36569ff5400" (UID: "7ab4e7d0-7531-4855-b771-f36569ff5400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.404957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.404939    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-envoy" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.405320 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.405303    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-kube-api-access-jph7t" pod "a12fcf30-ae85-4ba7-8228-768491bd5ee3" (UID: "a12fcf30-ae85-4ba7-8228-768491bd5ee3"). InnerVolumeSpecName "kube-api-access-jph7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:24.498299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.498268    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:46:24.498546 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.498358    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:46:24.504769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504742    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504769 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504769    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504780    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504789    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab4e7d0-7531-4855-b771-f36569ff5400-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504799    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8n2\" (UniqueName: \"kubernetes.io/projected/7ab4e7d0-7531-4855-b771-f36569ff5400-kube-api-access-cj8n2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504812    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504826    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jph7t\" (UniqueName: \"kubernetes.io/projected/a12fcf30-ae85-4ba7-8228-768491bd5ee3-kube-api-access-jph7t\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504840    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504854    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.504946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.504868    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a12fcf30-ae85-4ba7-8228-768491bd5ee3-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.575309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.571155    5423 generic.go:356] "Generic (PLEG): container finished" podID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerID="1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f" exitCode=0
Nov 01 14:46:24.575309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.571219    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"
Nov 01 14:46:24.575309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.571236    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" event={"ID":"a12fcf30-ae85-4ba7-8228-768491bd5ee3","Type":"ContainerDied","Data":"1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f"}
Nov 01 14:46:24.575309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.571312    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx" event={"ID":"a12fcf30-ae85-4ba7-8228-768491bd5ee3","Type":"ContainerDied","Data":"d786d94456f264f7aac2b6925c5858a3610d74d1804e3ba72fa858254c6c1fea"}
Nov 01 14:46:24.575309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.571355    5423 scope.go:122] "RemoveContainer" containerID="1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f"
Nov 01 14:46:24.592507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.592475    5423 generic.go:356] "Generic (PLEG): container finished" podID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerID="f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce" exitCode=0
Nov 01 14:46:24.593396 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.593361    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"
Nov 01 14:46:24.594600 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.594573    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" event={"ID":"7ab4e7d0-7531-4855-b771-f36569ff5400","Type":"ContainerDied","Data":"f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce"}
Nov 01 14:46:24.598074 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.598030    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn" event={"ID":"7ab4e7d0-7531-4855-b771-f36569ff5400","Type":"ContainerDied","Data":"f4187a81e9fa1e6282eb37948d87431c11971ed3d8e62083de8d34a9debd3739"}
Nov 01 14:46:24.607758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.607498    5423 scope.go:122] "RemoveContainer" containerID="1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f"
Nov 01 14:46:24.609158 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:46:24.609036    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f\": container with ID starting with 1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f not found: ID does not exist" containerID="1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f"
Nov 01 14:46:24.609255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.609185    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f"} err="failed to get container status \"1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f\": rpc error: code = NotFound desc = could not find container \"1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f\": container with ID starting with 1d2dd1cbe3cb41ba9d55381f9f8ebc7a799d5d064df92500a9758d4db676b01f not found: ID does not exist"
Nov 01 14:46:24.609255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.609214    5423 scope.go:122] "RemoveContainer" containerID="f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce"
Nov 01 14:46:24.631250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.631223    5423 scope.go:122] "RemoveContainer" containerID="2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9"
Nov 01 14:46:24.644434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.644408    5423 scope.go:122] "RemoveContainer" containerID="f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce"
Nov 01 14:46:24.644825 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:46:24.644796    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce\": container with ID starting with f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce not found: ID does not exist" containerID="f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce"
Nov 01 14:46:24.644892 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.644838    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce"} err="failed to get container status \"f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce\": rpc error: code = NotFound desc = could not find container \"f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce\": container with ID starting with f12940a76d78f1652cee2a560ada9e95b4f41c7cc00e205335271807420cfdce not found: ID does not exist"
Nov 01 14:46:24.644892 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.644866    5423 scope.go:122] "RemoveContainer" containerID="2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9"
Nov 01 14:46:24.645159 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:46:24.645134    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9\": container with ID starting with 2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9 not found: ID does not exist" containerID="2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9"
Nov 01 14:46:24.645234 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.645167    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9"} err="failed to get container status \"2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9\": rpc error: code = NotFound desc = could not find container \"2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9\": container with ID starting with 2a4de4043b1af7e3e21e16494a562afa747f4f10897a65b87c2d81202f455cf9 not found: ID does not exist"
Nov 01 14:46:24.703630 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.703597    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:46:24.977806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.977727    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"]
Nov 01 14:46:24.981327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:24.981301    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-tw9bn"]
Nov 01 14:46:25.092339 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:25.092310    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"]
Nov 01 14:46:25.096129 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:25.096105    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-cr8rx"]
Nov 01 14:46:25.961247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:25.961215    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" path="/var/lib/kubelet/pods/7ab4e7d0-7531-4855-b771-f36569ff5400/volumes"
Nov 01 14:46:25.961906 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:25.961884    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" path="/var/lib/kubelet/pods/a12fcf30-ae85-4ba7-8228-768491bd5ee3/volumes"
Nov 01 14:46:29.618407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.618209    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"]
Nov 01 14:46:29.618967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.618890    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy"
Nov 01 14:46:29.618967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.618904    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy"
Nov 01 14:46:29.618967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.618947    5423 cpu_manager.go:414] "containerMap: removing container" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy"
Nov 01 14:46:29.618967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.618953    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy"
Nov 01 14:46:29.619176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.619074    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy"
Nov 01 14:46:29.619176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.619087    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12fcf30-ae85-4ba7-8228-768491bd5ee3" containerName="istio-proxy"
Nov 01 14:46:29.619176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.619095    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy"
Nov 01 14:46:29.626760 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.626566    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:29.673958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.673923    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"]
Nov 01 14:46:29.680456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.680428    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-71e39a\""
Nov 01 14:46:29.680580 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.680472    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-71e39a-dockercfg-wx2nk\""
Nov 01 14:46:29.714561 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.714531    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abbebb1-3e50-414c-b641-c9d66e824ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf\" (UID: \"3abbebb1-3e50-414c-b641-c9d66e824ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:29.714719 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.714702    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqhbd\" (UniqueName: \"kubernetes.io/projected/3abbebb1-3e50-414c-b641-c9d66e824ce7-kube-api-access-sqhbd\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf\" (UID: \"3abbebb1-3e50-414c-b641-c9d66e824ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:29.840405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.818129    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqhbd\" (UniqueName: \"kubernetes.io/projected/3abbebb1-3e50-414c-b641-c9d66e824ce7-kube-api-access-sqhbd\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf\" (UID: \"3abbebb1-3e50-414c-b641-c9d66e824ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:29.840405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.818297    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abbebb1-3e50-414c-b641-c9d66e824ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf\" (UID: \"3abbebb1-3e50-414c-b641-c9d66e824ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:29.840405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.819254    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abbebb1-3e50-414c-b641-c9d66e824ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf\" (UID: \"3abbebb1-3e50-414c-b641-c9d66e824ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:29.897250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.894967    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqhbd\" (UniqueName: \"kubernetes.io/projected/3abbebb1-3e50-414c-b641-c9d66e824ce7-kube-api-access-sqhbd\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf\" (UID: \"3abbebb1-3e50-414c-b641-c9d66e824ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:29.945218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.945189    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:29.976553 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.976521    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"]
Nov 01 14:46:29.977443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.977419    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy"
Nov 01 14:46:29.977443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.977442    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab4e7d0-7531-4855-b771-f36569ff5400" containerName="istio-proxy"
Nov 01 14:46:29.981661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:29.981634    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:30.022937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.022003    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d594124e-2f15-4803-bcab-050ffddbea69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb\" (UID: \"d594124e-2f15-4803-bcab-050ffddbea69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:30.022937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.022819    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbh4\" (UniqueName: \"kubernetes.io/projected/d594124e-2f15-4803-bcab-050ffddbea69-kube-api-access-spbh4\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb\" (UID: \"d594124e-2f15-4803-bcab-050ffddbea69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:30.032357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.032335    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"]
Nov 01 14:46:30.035079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.035059    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-34d384\""
Nov 01 14:46:30.035256 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.035237    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-34d384-dockercfg-wtjdw\""
Nov 01 14:46:30.124887 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.124852    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d594124e-2f15-4803-bcab-050ffddbea69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb\" (UID: \"d594124e-2f15-4803-bcab-050ffddbea69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:30.125035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.124935    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbh4\" (UniqueName: \"kubernetes.io/projected/d594124e-2f15-4803-bcab-050ffddbea69-kube-api-access-spbh4\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb\" (UID: \"d594124e-2f15-4803-bcab-050ffddbea69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:30.125573 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.125554    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d594124e-2f15-4803-bcab-050ffddbea69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb\" (UID: \"d594124e-2f15-4803-bcab-050ffddbea69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:30.182939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.182872    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbh4\" (UniqueName: \"kubernetes.io/projected/d594124e-2f15-4803-bcab-050ffddbea69-kube-api-access-spbh4\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb\" (UID: \"d594124e-2f15-4803-bcab-050ffddbea69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:30.298223 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.298194    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:30.317300 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:30.317261    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abbebb1_3e50_414c_b641_c9d66e824ce7.slice/crio-c3f212279c8c727eede68e9573a8f8b0b46244994f3b018921cbda6d90dccf67 WatchSource:0}: Error finding container c3f212279c8c727eede68e9573a8f8b0b46244994f3b018921cbda6d90dccf67: Status 404 returned error can't find the container with id c3f212279c8c727eede68e9573a8f8b0b46244994f3b018921cbda6d90dccf67
Nov 01 14:46:30.320704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.320676    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"]
Nov 01 14:46:30.413160 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.413128    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"]
Nov 01 14:46:30.417187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.417162    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:30.427578 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.427552    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmk7t\" (UniqueName: \"kubernetes.io/projected/78378ebb-07db-47e6-8f2a-bc89d6c3d838-kube-api-access-cmk7t\") pod \"payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l\" (UID: \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:30.427797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.427775    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78378ebb-07db-47e6-8f2a-bc89d6c3d838-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l\" (UID: \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:30.471296 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.471235    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"]
Nov 01 14:46:30.529017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.528986    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmk7t\" (UniqueName: \"kubernetes.io/projected/78378ebb-07db-47e6-8f2a-bc89d6c3d838-kube-api-access-cmk7t\") pod \"payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l\" (UID: \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:30.529267 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.529102    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78378ebb-07db-47e6-8f2a-bc89d6c3d838-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l\" (UID: \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:30.529924 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.529904    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78378ebb-07db-47e6-8f2a-bc89d6c3d838-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l\" (UID: \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:30.587918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.587885    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmk7t\" (UniqueName: \"kubernetes.io/projected/78378ebb-07db-47e6-8f2a-bc89d6c3d838-kube-api-access-cmk7t\") pod \"payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l\" (UID: \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:30.637253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.637219    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf" event={"ID":"3abbebb1-3e50-414c-b641-c9d66e824ce7","Type":"ContainerStarted","Data":"6646508937ea9372a19c7aba9f351cd871d936b2041c268449686e51489984ff"}
Nov 01 14:46:30.637713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.637271    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf" event={"ID":"3abbebb1-3e50-414c-b641-c9d66e824ce7","Type":"ContainerStarted","Data":"c3f212279c8c727eede68e9573a8f8b0b46244994f3b018921cbda6d90dccf67"}
Nov 01 14:46:30.637713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.637340    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:30.671178 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:30.671148    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd594124e_2f15_4803_bcab_050ffddbea69.slice/crio-19d6fd5de720a92eb37a4c9e1f1333d6dc5e7973d035205adb285fa7b589e228 WatchSource:0}: Error finding container 19d6fd5de720a92eb37a4c9e1f1333d6dc5e7973d035205adb285fa7b589e228: Status 404 returned error can't find the container with id 19d6fd5de720a92eb37a4c9e1f1333d6dc5e7973d035205adb285fa7b589e228
Nov 01 14:46:30.673925 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.673900    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"]
Nov 01 14:46:30.732947 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.732925    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:30.756445 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:30.756364    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf" podStartSLOduration=1.756349977 podStartE2EDuration="1.756349977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:30.755653806 +0000 UTC m=+1730.921139104" watchObservedRunningTime="2026-11-01 14:46:30.756349977 +0000 UTC m=+1730.921835274"
Nov 01 14:46:31.102323 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:31.102288    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78378ebb_07db_47e6_8f2a_bc89d6c3d838.slice/crio-13e7801c4f915cf194055b22ef18990140f15a4563c426237cf06efbafba0d15 WatchSource:0}: Error finding container 13e7801c4f915cf194055b22ef18990140f15a4563c426237cf06efbafba0d15: Status 404 returned error can't find the container with id 13e7801c4f915cf194055b22ef18990140f15a4563c426237cf06efbafba0d15
Nov 01 14:46:31.105837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:31.105814    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"]
Nov 01 14:46:31.645241 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:31.645201    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb" event={"ID":"d594124e-2f15-4803-bcab-050ffddbea69","Type":"ContainerStarted","Data":"ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34"}
Nov 01 14:46:31.645726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:31.645421    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:31.645726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:31.645473    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb" event={"ID":"d594124e-2f15-4803-bcab-050ffddbea69","Type":"ContainerStarted","Data":"19d6fd5de720a92eb37a4c9e1f1333d6dc5e7973d035205adb285fa7b589e228"}
Nov 01 14:46:31.646900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:31.646871    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l" event={"ID":"78378ebb-07db-47e6-8f2a-bc89d6c3d838","Type":"ContainerStarted","Data":"0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c"}
Nov 01 14:46:31.647049 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:31.646911    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l" event={"ID":"78378ebb-07db-47e6-8f2a-bc89d6c3d838","Type":"ContainerStarted","Data":"13e7801c4f915cf194055b22ef18990140f15a4563c426237cf06efbafba0d15"}
Nov 01 14:46:31.760968 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:31.760909    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb" podStartSLOduration=2.760895873 podStartE2EDuration="2.760895873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:31.760473586 +0000 UTC m=+1731.925958886" watchObservedRunningTime="2026-11-01 14:46:31.760895873 +0000 UTC m=+1731.926381170"
Nov 01 14:46:32.652220 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:32.652188    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:35.149583 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:46:35.149542    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:46:41.649425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:41.649367    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:46:41.766313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:41.766254    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l" podStartSLOduration=11.766239571 podStartE2EDuration="11.766239571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:31.88989739 +0000 UTC m=+1732.055382687" watchObservedRunningTime="2026-11-01 14:46:41.766239571 +0000 UTC m=+1741.931724868"
Nov 01 14:46:42.655761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:42.655726    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:46:43.659380 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:43.659349    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:46:43.674253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:43.674218    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"]
Nov 01 14:46:43.674475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:43.674451    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy" containerID="cri-o://9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789" gracePeriod=30
Nov 01 14:46:48.946060 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:48.946001    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:46:49.070280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070249    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-istio-token\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-istio-token\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070306    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-envoy\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070340    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40036455-b3f2-42fa-9aec-c29413f37802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40036455-b3f2-42fa-9aec-c29413f37802-istio-podinfo\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070373    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-socket\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070439    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-credential-socket\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070478    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-data\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070515    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-kube-api-access-xz69p\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-kube-api-access-xz69p\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070543    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40036455-b3f2-42fa-9aec-c29413f37802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40036455-b3f2-42fa-9aec-c29413f37802-istiod-ca-cert\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070640    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-certs\") pod \"40036455-b3f2-42fa-9aec-c29413f37802\" (UID: \"40036455-b3f2-42fa-9aec-c29413f37802\") "
Nov 01 14:46:49.070896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070794    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-credential-socket" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:49.070896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.070803    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-socket" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:49.071161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.071108    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-certs" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:49.071161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.071118    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40036455-b3f2-42fa-9aec-c29413f37802-istiod-ca-cert" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:46:49.071316 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.071207    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-data" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:49.071375 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.071332    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.071375 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.071353    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.071375 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.071369    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.071617 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.071408    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40036455-b3f2-42fa-9aec-c29413f37802-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.072740 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.072715    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-istio-token" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:49.072865 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.072847    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-kube-api-access-xz69p" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "kube-api-access-xz69p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:49.073062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.073046    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-envoy" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:49.073122 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.073087    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40036455-b3f2-42fa-9aec-c29413f37802-istio-podinfo" pod "40036455-b3f2-42fa-9aec-c29413f37802" (UID: "40036455-b3f2-42fa-9aec-c29413f37802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:46:49.173032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.172997    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.173032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.173035    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.173284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.173060    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40036455-b3f2-42fa-9aec-c29413f37802-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.173284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.173079    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40036455-b3f2-42fa-9aec-c29413f37802-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.173284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.173096    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz69p\" (UniqueName: \"kubernetes.io/projected/40036455-b3f2-42fa-9aec-c29413f37802-kube-api-access-xz69p\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:49.897183 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.897145    5423 generic.go:356] "Generic (PLEG): container finished" podID="40036455-b3f2-42fa-9aec-c29413f37802" containerID="9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789" exitCode=0
Nov 01 14:46:49.897641 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.897610    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" event={"ID":"40036455-b3f2-42fa-9aec-c29413f37802","Type":"ContainerDied","Data":"9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789"}
Nov 01 14:46:49.897892 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.897870    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq" event={"ID":"40036455-b3f2-42fa-9aec-c29413f37802","Type":"ContainerDied","Data":"d817a27653e10460061ed5ae81c638ae343d1452fe679c5b7f7cd0c98913d44c"}
Nov 01 14:46:49.898042 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.898026    5423 scope.go:122] "RemoveContainer" containerID="9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789"
Nov 01 14:46:49.902419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.898544    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"
Nov 01 14:46:49.923294 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.923268    5423 scope.go:122] "RemoveContainer" containerID="4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0"
Nov 01 14:46:49.970369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.970331    5423 scope.go:122] "RemoveContainer" containerID="9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789"
Nov 01 14:46:49.971112 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:46:49.970811    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789\": container with ID starting with 9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789 not found: ID does not exist" containerID="9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789"
Nov 01 14:46:49.971112 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.970848    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789"} err="failed to get container status \"9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789\": rpc error: code = NotFound desc = could not find container \"9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789\": container with ID starting with 9ef256004c0bded3ecf46da93133461954dfdc9251bb72a486f7da270d079789 not found: ID does not exist"
Nov 01 14:46:49.971112 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.970872    5423 scope.go:122] "RemoveContainer" containerID="4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0"
Nov 01 14:46:49.971238 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:46:49.971195    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0\": container with ID starting with 4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0 not found: ID does not exist" containerID="4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0"
Nov 01 14:46:49.971238 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:49.971222    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0"} err="failed to get container status \"4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0\": rpc error: code = NotFound desc = could not find container \"4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0\": container with ID starting with 4eb198a2bf0f1174cbadcfcf91f3e41d577064acc771f9de691632e3861988c0 not found: ID does not exist"
Nov 01 14:46:50.137817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:50.137788    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"]
Nov 01 14:46:50.189860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:50.189796    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-5g2zq"]
Nov 01 14:46:51.960853 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:51.960818    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40036455-b3f2-42fa-9aec-c29413f37802" path="/var/lib/kubelet/pods/40036455-b3f2-42fa-9aec-c29413f37802/volumes"
Nov 01 14:46:53.622627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.622598    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"]
Nov 01 14:46:53.623176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.623161    5423 cpu_manager.go:414] "containerMap: removing container" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy"
Nov 01 14:46:53.623228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.623177    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy"
Nov 01 14:46:53.623228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.623202    5423 cpu_manager.go:414] "containerMap: removing container" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy"
Nov 01 14:46:53.623228 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.623207    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy"
Nov 01 14:46:53.623326 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.623282    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy"
Nov 01 14:46:53.627994 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.627969    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:53.679364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.679330    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"]
Nov 01 14:46:53.727997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.727964    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-25ce0b-dockercfg-nnltv\""
Nov 01 14:46:53.728146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.728008    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-25ce0b\""
Nov 01 14:46:53.799915 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.799889    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceda76db-81d2-4b69-abe2-5121ffe7d88b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm\" (UID: \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:53.800076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.800056    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsf67\" (UniqueName: \"kubernetes.io/projected/ceda76db-81d2-4b69-abe2-5121ffe7d88b-kube-api-access-zsf67\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm\" (UID: \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:53.901298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.901216    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsf67\" (UniqueName: \"kubernetes.io/projected/ceda76db-81d2-4b69-abe2-5121ffe7d88b-kube-api-access-zsf67\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm\" (UID: \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:53.901489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.901327    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceda76db-81d2-4b69-abe2-5121ffe7d88b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm\" (UID: \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:53.902047 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.902027    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceda76db-81d2-4b69-abe2-5121ffe7d88b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm\" (UID: \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:53.959262 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:53.959230    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsf67\" (UniqueName: \"kubernetes.io/projected/ceda76db-81d2-4b69-abe2-5121ffe7d88b-kube-api-access-zsf67\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm\" (UID: \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:54.240828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.240747    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:54.368045 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.368012    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"]
Nov 01 14:46:54.369111 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.369089    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="40036455-b3f2-42fa-9aec-c29413f37802" containerName="istio-proxy"
Nov 01 14:46:54.374094 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.374075    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:54.461019 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.460980    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"]
Nov 01 14:46:54.507904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.507836    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s\" (UID: \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:54.507904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.507883    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcvk\" (UniqueName: \"kubernetes.io/projected/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-kube-api-access-4mcvk\") pod \"payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s\" (UID: \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:54.609224 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.609191    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s\" (UID: \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:54.609377 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.609238    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcvk\" (UniqueName: \"kubernetes.io/projected/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-kube-api-access-4mcvk\") pod \"payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s\" (UID: \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:54.609897 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.609877    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s\" (UID: \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:54.665459 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:54.665418    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceda76db_81d2_4b69_abe2_5121ffe7d88b.slice/crio-9e8cc8235e8c8e2380a2df7b8a6c0bca1d4ad06c5f39c406658d6df1d60f805d WatchSource:0}: Error finding container 9e8cc8235e8c8e2380a2df7b8a6c0bca1d4ad06c5f39c406658d6df1d60f805d: Status 404 returned error can't find the container with id 9e8cc8235e8c8e2380a2df7b8a6c0bca1d4ad06c5f39c406658d6df1d60f805d
Nov 01 14:46:54.668815 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.668702    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"]
Nov 01 14:46:54.668880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.668810    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcvk\" (UniqueName: \"kubernetes.io/projected/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-kube-api-access-4mcvk\") pod \"payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s\" (UID: \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:54.688586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.688564    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:54.966370 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.966332    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm" event={"ID":"ceda76db-81d2-4b69-abe2-5121ffe7d88b","Type":"ContainerStarted","Data":"2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6"}
Nov 01 14:46:54.966559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.966380    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm" event={"ID":"ceda76db-81d2-4b69-abe2-5121ffe7d88b","Type":"ContainerStarted","Data":"9e8cc8235e8c8e2380a2df7b8a6c0bca1d4ad06c5f39c406658d6df1d60f805d"}
Nov 01 14:46:54.966559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:54.966435    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:46:55.066668 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:55.066632    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a92e04_f8d0_4bc8_9fcd_33dd819603f0.slice/crio-31820f3b681b2869d929d8d39cf454c1f3606ef93ccd32ba82d4c30889279504 WatchSource:0}: Error finding container 31820f3b681b2869d929d8d39cf454c1f3606ef93ccd32ba82d4c30889279504: Status 404 returned error can't find the container with id 31820f3b681b2869d929d8d39cf454c1f3606ef93ccd32ba82d4c30889279504
Nov 01 14:46:55.070203 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:55.070171    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"]
Nov 01 14:46:55.082915 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:55.082868    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm" podStartSLOduration=2.082851657 podStartE2EDuration="2.082851657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:55.08278857 +0000 UTC m=+1755.248273878" watchObservedRunningTime="2026-11-01 14:46:55.082851657 +0000 UTC m=+1755.248336956"
Nov 01 14:46:55.974984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:55.974945    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s" event={"ID":"65a92e04-f8d0-4bc8-9fcd-33dd819603f0","Type":"ContainerStarted","Data":"de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81"}
Nov 01 14:46:55.975453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:55.975007    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s" event={"ID":"65a92e04-f8d0-4bc8-9fcd-33dd819603f0","Type":"ContainerStarted","Data":"31820f3b681b2869d929d8d39cf454c1f3606ef93ccd32ba82d4c30889279504"}
Nov 01 14:46:55.975453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:55.975038    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:46:56.093295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:56.093241    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s" podStartSLOduration=2.093225954 podStartE2EDuration="2.093225954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:56.091892235 +0000 UTC m=+1756.257377544" watchObservedRunningTime="2026-11-01 14:46:56.093225954 +0000 UTC m=+1756.258711251"
Nov 01 14:46:58.071271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.071229    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"]
Nov 01 14:46:58.076504 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.076479    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"]
Nov 01 14:46:58.076775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.076723    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:58.181684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.181653    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7c335a\""
Nov 01 14:46:58.181684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.181654    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7c335a-dockercfg-bc8hl\""
Nov 01 14:46:58.227024 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.226997    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405734c6-9251-48e5-ae92-8fad2d3c1c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs\" (UID: \"405734c6-9251-48e5-ae92-8fad2d3c1c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:58.227166 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.227031    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhww\" (UniqueName: \"kubernetes.io/projected/405734c6-9251-48e5-ae92-8fad2d3c1c64-kube-api-access-rhhww\") pod \"payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs\" (UID: \"405734c6-9251-48e5-ae92-8fad2d3c1c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:58.327852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.327765    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405734c6-9251-48e5-ae92-8fad2d3c1c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs\" (UID: \"405734c6-9251-48e5-ae92-8fad2d3c1c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:58.327852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.327809    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhww\" (UniqueName: \"kubernetes.io/projected/405734c6-9251-48e5-ae92-8fad2d3c1c64-kube-api-access-rhhww\") pod \"payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs\" (UID: \"405734c6-9251-48e5-ae92-8fad2d3c1c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:58.328492 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.328471    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405734c6-9251-48e5-ae92-8fad2d3c1c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs\" (UID: \"405734c6-9251-48e5-ae92-8fad2d3c1c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:58.385829 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.385795    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhww\" (UniqueName: \"kubernetes.io/projected/405734c6-9251-48e5-ae92-8fad2d3c1c64-kube-api-access-rhhww\") pod \"payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs\" (UID: \"405734c6-9251-48e5-ae92-8fad2d3c1c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:58.392633 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.392603    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:58.759253 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:46:58.759224    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405734c6_9251_48e5_ae92_8fad2d3c1c64.slice/crio-0ca9d4fa26a9ac0691762b2ebfe5a9c4b6338db5622a5ea6d7fc0866216bd87a WatchSource:0}: Error finding container 0ca9d4fa26a9ac0691762b2ebfe5a9c4b6338db5622a5ea6d7fc0866216bd87a: Status 404 returned error can't find the container with id 0ca9d4fa26a9ac0691762b2ebfe5a9c4b6338db5622a5ea6d7fc0866216bd87a
Nov 01 14:46:58.762575 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:58.762547    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"]
Nov 01 14:46:59.052861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:59.052771    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs" event={"ID":"405734c6-9251-48e5-ae92-8fad2d3c1c64","Type":"ContainerStarted","Data":"20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c"}
Nov 01 14:46:59.052861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:59.052829    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:46:59.052861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:59.052847    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs" event={"ID":"405734c6-9251-48e5-ae92-8fad2d3c1c64","Type":"ContainerStarted","Data":"0ca9d4fa26a9ac0691762b2ebfe5a9c4b6338db5622a5ea6d7fc0866216bd87a"}
Nov 01 14:46:59.221941 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:59.221892    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs" podStartSLOduration=2.221875906 podStartE2EDuration="2.221875906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:59.169061526 +0000 UTC m=+1759.334546823" watchObservedRunningTime="2026-11-01 14:46:59.221875906 +0000 UTC m=+1759.387361205"
Nov 01 14:46:59.979075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:59.979038    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"]
Nov 01 14:46:59.985897 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:46:59.985683    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:00.035711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.035677    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"]
Nov 01 14:47:00.054776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.054742    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8s9\" (UniqueName: \"kubernetes.io/projected/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-kube-api-access-tq8s9\") pod \"payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml\" (UID: \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:00.055218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.054813    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml\" (UID: \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:00.084134 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.084099    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cbac83\""
Nov 01 14:47:00.084134 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.084108    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cbac83-dockercfg-m6l7p\""
Nov 01 14:47:00.156219 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.156191    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8s9\" (UniqueName: \"kubernetes.io/projected/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-kube-api-access-tq8s9\") pod \"payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml\" (UID: \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:00.156342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.156232    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml\" (UID: \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:00.157029 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.157008    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml\" (UID: \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:00.214253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.214226    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8s9\" (UniqueName: \"kubernetes.io/projected/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-kube-api-access-tq8s9\") pod \"payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml\" (UID: \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:00.301602 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.301530    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:00.669095 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:47:00.669065    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f3bc07_4b36_42ba_8efb_2a03ebc4ce42.slice/crio-c0fe9c8159e2de10fcb1c00f1b7b3d384b0a8c6fbb019fd4b631f279b051d4d2 WatchSource:0}: Error finding container c0fe9c8159e2de10fcb1c00f1b7b3d384b0a8c6fbb019fd4b631f279b051d4d2: Status 404 returned error can't find the container with id c0fe9c8159e2de10fcb1c00f1b7b3d384b0a8c6fbb019fd4b631f279b051d4d2
Nov 01 14:47:00.672565 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:00.672537    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"]
Nov 01 14:47:01.069355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:01.069324    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml" event={"ID":"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42","Type":"ContainerStarted","Data":"38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed"}
Nov 01 14:47:01.069543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:01.069378    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml" event={"ID":"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42","Type":"ContainerStarted","Data":"c0fe9c8159e2de10fcb1c00f1b7b3d384b0a8c6fbb019fd4b631f279b051d4d2"}
Nov 01 14:47:01.069543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:01.069428    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:01.189217 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:01.189161    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml" podStartSLOduration=2.189147785 podStartE2EDuration="2.189147785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:47:01.187963136 +0000 UTC m=+1761.353448437" watchObservedRunningTime="2026-11-01 14:47:01.189147785 +0000 UTC m=+1761.354633118"
Nov 01 14:47:05.160445 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:47:05.160402    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:47:05.977197 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:05.977171    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:47:06.987430 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:06.987382    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:47:10.065448 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:10.065410    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:47:12.076914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:12.076886    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:47:24.297470 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.297435    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"]
Nov 01 14:47:24.302909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.302888    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.348685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.348658    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"]
Nov 01 14:47:24.352734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.352717    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.400141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.400114    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"]
Nov 01 14:47:24.416434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.416378    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6dtj\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-kube-api-access-v6dtj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.416551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.416496    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.416551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.416534    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.416681 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.416662    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19eaa467-1361-4dac-acde-767aad1eadd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.416717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.416698    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.416818 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.416792    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.416866 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.416849    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.417117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.417067    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19eaa467-1361-4dac-acde-767aad1eadd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.417195 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.417176    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.445768 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.445741    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"]
Nov 01 14:47:24.522202 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.522157    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.522399 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.522231    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.522399 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.522271    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8jd4\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-kube-api-access-f8jd4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.522399 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.522307    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.522399 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.522342    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6dtj\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-kube-api-access-v6dtj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.522399 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.522371    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.522680 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.522423    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.523086 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523060    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19eaa467-1361-4dac-acde-767aad1eadd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.523193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523097    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.523193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523118    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.523193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523118    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.523193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523139    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.523193 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523067    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.523474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523259    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.523474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523288    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.523474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523323    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.523474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523357    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.523474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523413    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.523474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523460    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.523780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523556    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19eaa467-1361-4dac-acde-767aad1eadd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.523780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.523687    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.524417 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.524374    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19eaa467-1361-4dac-acde-767aad1eadd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.526474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.526444    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.532636 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.532614    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.533992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.533974    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19eaa467-1361-4dac-acde-767aad1eadd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.584157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.584135    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.584552 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.584528    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6dtj\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-kube-api-access-v6dtj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l8f9k\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.618149 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.618126    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:24.624750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.624728    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.624863 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.624771    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.624863 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.624811    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.624863 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.624850    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.625041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.624957    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.625041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.625007    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8jd4\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-kube-api-access-f8jd4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.625041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.625030    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.625207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.625127    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.625207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.625151    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.625207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.625177    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.625569 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.625543    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.625985 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.625954    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.626076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.625994    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.626299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.626276    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.627593 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.627573    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.627914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.627895    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.690422 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.689567    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8jd4\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-kube-api-access-f8jd4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.691933 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.691905    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xkd92\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:24.965961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:24.965884    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:25.038327 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.038299    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"]
Nov 01 14:47:25.043355 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:47:25.043328    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19eaa467_1361_4dac_acde_767aad1eadd0.slice/crio-632bb254108c159814af419d6076f6bafedd27aaaeb8b84ef29ad5f777977111 WatchSource:0}: Error finding container 632bb254108c159814af419d6076f6bafedd27aaaeb8b84ef29ad5f777977111: Status 404 returned error can't find the container with id 632bb254108c159814af419d6076f6bafedd27aaaeb8b84ef29ad5f777977111
Nov 01 14:47:25.045867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.045836    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:25.045972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.045912    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:25.046041 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.046023    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:25.227516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.227435    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" event={"ID":"19eaa467-1361-4dac-acde-767aad1eadd0","Type":"ContainerStarted","Data":"013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad"}
Nov 01 14:47:25.227516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.227493    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" event={"ID":"19eaa467-1361-4dac-acde-767aad1eadd0","Type":"ContainerStarted","Data":"632bb254108c159814af419d6076f6bafedd27aaaeb8b84ef29ad5f777977111"}
Nov 01 14:47:25.335360 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:47:25.335319    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8cf631_16c6_4ed6_b1ad_5ff4e79ccab5.slice/crio-1a8e13c4a7c02144709e89b6bb7e57e21952cee91fefe123957266d24e3b8c8e WatchSource:0}: Error finding container 1a8e13c4a7c02144709e89b6bb7e57e21952cee91fefe123957266d24e3b8c8e: Status 404 returned error can't find the container with id 1a8e13c4a7c02144709e89b6bb7e57e21952cee91fefe123957266d24e3b8c8e
Nov 01 14:47:25.338260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.338224    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:25.338357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.338290    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:25.338357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.338317    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:25.340152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.340129    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"]
Nov 01 14:47:25.347997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.347954    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podStartSLOduration=1.347942787 podStartE2EDuration="1.347942787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:47:25.346406817 +0000 UTC m=+1785.511892112" watchObservedRunningTime="2026-11-01 14:47:25.347942787 +0000 UTC m=+1785.513428169"
Nov 01 14:47:25.618570 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:25.618526    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:26.254063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:26.253942    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" event={"ID":"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5","Type":"ContainerStarted","Data":"f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb"}
Nov 01 14:47:26.254063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:26.254023    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" event={"ID":"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5","Type":"ContainerStarted","Data":"1a8e13c4a7c02144709e89b6bb7e57e21952cee91fefe123957266d24e3b8c8e"}
Nov 01 14:47:26.431409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:26.431229    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podStartSLOduration=2.431209734 podStartE2EDuration="2.431209734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:47:26.427974683 +0000 UTC m=+1786.593459982" watchObservedRunningTime="2026-11-01 14:47:26.431209734 +0000 UTC m=+1786.596695033"
Nov 01 14:47:26.632556 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:26.632513    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l8f9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 01 14:47:26.632836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:26.632797    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 01 14:47:26.967039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:26.966957    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:26.968775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:26.968743    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xkd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 14:47:26.968898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:26.968797    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 14:47:27.619410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:27.618781    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l8f9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 01 14:47:27.619410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:27.618845    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 01 14:47:27.966956 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:27.966870    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xkd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 14:47:27.966956 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:27.966921    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 14:47:28.623409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:28.618737    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l8f9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 01 14:47:28.623409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:28.618797    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 01 14:47:28.967002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:28.966913    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xkd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 14:47:28.967002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:28.966977    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 14:47:29.619419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:29.618720    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l8f9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 01 14:47:29.619419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:29.618784    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 01 14:47:29.966280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:29.966198    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xkd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 14:47:29.966280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:29.966260    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 14:47:30.649957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:30.647569    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l8f9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 01 14:47:30.649957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:30.647655    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 01 14:47:30.968064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:30.967979    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xkd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 14:47:30.968703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:30.968657    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 14:47:31.619021 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:31.618982    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l8f9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 01 14:47:31.619168 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:31.619042    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 01 14:47:31.966846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:31.966768    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xkd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 14:47:31.966846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:31.966818    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 14:47:32.619306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:32.619270    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l8f9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Nov 01 14:47:32.619705 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:32.619334    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Nov 01 14:47:32.966584 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:32.966490    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xkd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 14:47:32.966584 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:32.966563    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 14:47:33.759799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:33.759761    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:33.761564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:33.761379    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:33.763249 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:33.763228    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:47:34.095950 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:34.095894    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:34.097564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:34.097536    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:34.098538 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:34.098515    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 14:47:35.150454 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:47:35.150413    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:47:37.129642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:37.129609    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:47:39.420410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:39.419572    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"]
Nov 01 14:47:39.420410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:39.419820    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4" podUID="e392998c-8d13-4a07-9020-c9bf3292a2c4" containerName="maas-api" containerID="cri-o://51e8e408c654096e1d6ab74216a74f2a7e0115abc0eade4f515cfa72d750f94f" gracePeriod=30
Nov 01 14:47:41.004461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:40.986512    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"]
Nov 01 14:47:41.004461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:40.990514    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn" podUID="bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0" containerName="payload-processing" containerID="cri-o://f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19" gracePeriod=30
Nov 01 14:47:41.092444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:41.084644    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"]
Nov 01 14:47:41.208406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:41.169644    5423 generic.go:356] "Generic (PLEG): container finished" podID="e392998c-8d13-4a07-9020-c9bf3292a2c4" containerID="51e8e408c654096e1d6ab74216a74f2a7e0115abc0eade4f515cfa72d750f94f" exitCode=0
Nov 01 14:47:41.208406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:41.201468    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4" event={"ID":"e392998c-8d13-4a07-9020-c9bf3292a2c4","Type":"ContainerDied","Data":"51e8e408c654096e1d6ab74216a74f2a7e0115abc0eade4f515cfa72d750f94f"}
Nov 01 14:47:41.208406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:41.202120    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf" podUID="09d974fc-18ae-41ee-8bac-5e22365f1cd6" containerName="payload-pre-processing" containerID="cri-o://83cb4fd12b66107b4efd4b22155e8fe8db401476faed771faba1afb4d88f7f4e" gracePeriod=30
Nov 01 14:47:41.668707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:41.667473    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:47:41.840446 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:41.837968    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:47:42.319411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.293122    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:47:42.350290 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:47:42.350227    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:47:42.359961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.359930    5423 generic.go:356] "Generic (PLEG): container finished" podID="09d974fc-18ae-41ee-8bac-5e22365f1cd6" containerID="83cb4fd12b66107b4efd4b22155e8fe8db401476faed771faba1afb4d88f7f4e" exitCode=0
Nov 01 14:47:42.360140 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.360026    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf" event={"ID":"09d974fc-18ae-41ee-8bac-5e22365f1cd6","Type":"ContainerDied","Data":"83cb4fd12b66107b4efd4b22155e8fe8db401476faed771faba1afb4d88f7f4e"}
Nov 01 14:47:42.368069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.368033    5423 generic.go:356] "Generic (PLEG): container finished" podID="bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0" containerID="f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19" exitCode=0
Nov 01 14:47:42.368225 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.368121    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn" event={"ID":"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0","Type":"ContainerDied","Data":"f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19"}
Nov 01 14:47:42.375117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.375085    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4" event={"ID":"e392998c-8d13-4a07-9020-c9bf3292a2c4","Type":"ContainerDied","Data":"a46716ff3c1604ebf276658ce58f083327ed1fb316ce8f596194662dd2e96dae"}
Nov 01 14:47:42.375253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.375152    5423 scope.go:122] "RemoveContainer" containerID="51e8e408c654096e1d6ab74216a74f2a7e0115abc0eade4f515cfa72d750f94f"
Nov 01 14:47:42.375448 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.375431    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"
Nov 01 14:47:42.384073 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.384051    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:47:42.388665 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.388638    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-maas-api-tls\") pod \"e392998c-8d13-4a07-9020-c9bf3292a2c4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") "
Nov 01 14:47:42.388900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.388879    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e392998c-8d13-4a07-9020-c9bf3292a2c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e392998c-8d13-4a07-9020-c9bf3292a2c4-openshift-service-ca\") pod \"e392998c-8d13-4a07-9020-c9bf3292a2c4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") "
Nov 01 14:47:42.388977 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.388941    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e392998c-8d13-4a07-9020-c9bf3292a2c4-kube-api-access-96nwk\" (UniqueName: \"kubernetes.io/projected/e392998c-8d13-4a07-9020-c9bf3292a2c4-kube-api-access-96nwk\") pod \"e392998c-8d13-4a07-9020-c9bf3292a2c4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") "
Nov 01 14:47:42.389082 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.389038    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-metrics-certs\") pod \"e392998c-8d13-4a07-9020-c9bf3292a2c4\" (UID: \"e392998c-8d13-4a07-9020-c9bf3292a2c4\") "
Nov 01 14:47:42.389841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.389818    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e392998c-8d13-4a07-9020-c9bf3292a2c4-openshift-service-ca" pod "e392998c-8d13-4a07-9020-c9bf3292a2c4" (UID: "e392998c-8d13-4a07-9020-c9bf3292a2c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:47:42.391052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.391033    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:47:42.391903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.391830    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e392998c-8d13-4a07-9020-c9bf3292a2c4-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:47:42.452973 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.452918    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-metrics-certs" pod "e392998c-8d13-4a07-9020-c9bf3292a2c4" (UID: "e392998c-8d13-4a07-9020-c9bf3292a2c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:47:42.454218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.454193    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-maas-api-tls" pod "e392998c-8d13-4a07-9020-c9bf3292a2c4" (UID: "e392998c-8d13-4a07-9020-c9bf3292a2c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:47:42.467661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.465822    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e392998c-8d13-4a07-9020-c9bf3292a2c4-kube-api-access-96nwk" pod "e392998c-8d13-4a07-9020-c9bf3292a2c4" (UID: "e392998c-8d13-4a07-9020-c9bf3292a2c4"). InnerVolumeSpecName "kube-api-access-96nwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:47:42.480270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.478646    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:47:42.492736 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.492712    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d974fc-18ae-41ee-8bac-5e22365f1cd6-kube-api-access-rdpv6\" (UniqueName: \"kubernetes.io/projected/09d974fc-18ae-41ee-8bac-5e22365f1cd6-kube-api-access-rdpv6\") pod \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\" (UID: \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\") "
Nov 01 14:47:42.492928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.492905    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d974fc-18ae-41ee-8bac-5e22365f1cd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d974fc-18ae-41ee-8bac-5e22365f1cd6-plugins-config-volume\") pod \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\" (UID: \"09d974fc-18ae-41ee-8bac-5e22365f1cd6\") "
Nov 01 14:47:42.493271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.493246    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d974fc-18ae-41ee-8bac-5e22365f1cd6-plugins-config-volume" pod "09d974fc-18ae-41ee-8bac-5e22365f1cd6" (UID: "09d974fc-18ae-41ee-8bac-5e22365f1cd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:47:42.493513 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.493498    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:47:42.493560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.493518    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d974fc-18ae-41ee-8bac-5e22365f1cd6-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:47:42.493560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.493528    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96nwk\" (UniqueName: \"kubernetes.io/projected/e392998c-8d13-4a07-9020-c9bf3292a2c4-kube-api-access-96nwk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:47:42.493560 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.493539    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e392998c-8d13-4a07-9020-c9bf3292a2c4-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:47:42.524678 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.524566    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d974fc-18ae-41ee-8bac-5e22365f1cd6-kube-api-access-rdpv6" pod "09d974fc-18ae-41ee-8bac-5e22365f1cd6" (UID: "09d974fc-18ae-41ee-8bac-5e22365f1cd6"). InnerVolumeSpecName "kube-api-access-rdpv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:47:42.555731 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.555705    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:47:42.622409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.594912    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-plugins-config-volume\") pod \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\" (UID: \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\") "
Nov 01 14:47:42.622409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.595016    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-kube-api-access-pcpvz\" (UniqueName: \"kubernetes.io/projected/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-kube-api-access-pcpvz\") pod \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\" (UID: \"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0\") "
Nov 01 14:47:42.622409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.595492    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpv6\" (UniqueName: \"kubernetes.io/projected/09d974fc-18ae-41ee-8bac-5e22365f1cd6-kube-api-access-rdpv6\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:47:42.622409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.596058    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-plugins-config-volume" pod "bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0" (UID: "bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:47:42.627268 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.627225    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-kube-api-access-pcpvz" pod "bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0" (UID: "bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0"). InnerVolumeSpecName "kube-api-access-pcpvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:47:42.697414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.697333    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcpvz\" (UniqueName: \"kubernetes.io/projected/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-kube-api-access-pcpvz\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:47:42.697414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.697366    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:47:42.851313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.851273    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"]
Nov 01 14:47:42.853689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:42.853666    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-18a879-688b5f9cbd-56js4"]
Nov 01 14:47:43.382437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.382349    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"
Nov 01 14:47:43.382437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.382354    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf" event={"ID":"09d974fc-18ae-41ee-8bac-5e22365f1cd6","Type":"ContainerDied","Data":"e30684ce58bff2554ef71abd2bf8b2429b73e03c816dd12d93b05dc1169042de"}
Nov 01 14:47:43.382923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.382452    5423 scope.go:122] "RemoveContainer" containerID="83cb4fd12b66107b4efd4b22155e8fe8db401476faed771faba1afb4d88f7f4e"
Nov 01 14:47:43.384247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.384230    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"
Nov 01 14:47:43.384368 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.384228    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn" event={"ID":"bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0","Type":"ContainerDied","Data":"3d347fba94ee8bd58f4b68830ee6d0b75186a2dce0eb00de64d1516f0b18b856"}
Nov 01 14:47:43.394076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.394058    5423 scope.go:122] "RemoveContainer" containerID="f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19"
Nov 01 14:47:43.550584 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.550552    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"]
Nov 01 14:47:43.624278 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.624243    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18a879-77bdb9f94c-kvlcf"]
Nov 01 14:47:43.675426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.675339    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"]
Nov 01 14:47:43.678627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.678601    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18a879-77c774496b-msgfn"]
Nov 01 14:47:43.964267 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.964186    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d974fc-18ae-41ee-8bac-5e22365f1cd6" path="/var/lib/kubelet/pods/09d974fc-18ae-41ee-8bac-5e22365f1cd6/volumes"
Nov 01 14:47:43.964725 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.964702    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0" path="/var/lib/kubelet/pods/bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0/volumes"
Nov 01 14:47:43.965158 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:43.965139    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e392998c-8d13-4a07-9020-c9bf3292a2c4" path="/var/lib/kubelet/pods/e392998c-8d13-4a07-9020-c9bf3292a2c4/volumes"
Nov 01 14:47:48.419443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:48.419378    5423 generic.go:356] "Generic (PLEG): container finished" podID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerID="baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8" exitCode=0
Nov 01 14:47:48.419864 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:48.419463    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" event={"ID":"0b08385e-e72e-4078-9dce-f385d0ba5ad3","Type":"ContainerDied","Data":"baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8"}
Nov 01 14:47:48.419911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:48.419891    5423 scope.go:122] "RemoveContainer" containerID="baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8"
Nov 01 14:47:48.421295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:48.421267    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:48.421371 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:48.421331    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:48.421371 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:48.421362    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:47:49.380752 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:47:49.380717    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:47:49.428279 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:49.428206    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" event={"ID":"0b08385e-e72e-4078-9dce-f385d0ba5ad3","Type":"ContainerStarted","Data":"bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c"}
Nov 01 14:47:49.913410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:49.909042    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:47:49.913410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:49.909182    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 14:47:49.913410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:49.909241    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 14:47:50.433495 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:50.433463    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:47:50.919533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:50.919498    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 14:47:50.919736 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:50.919578    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 14:47:51.915410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:51.912797    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 14:47:51.915410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:51.912860    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 14:47:52.370068 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:47:52.370029    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:47:52.463139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463106    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"]
Nov 01 14:47:52.463733 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463718    5423 cpu_manager.go:414] "containerMap: removing container" podUID="bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0" containerName="payload-processing"
Nov 01 14:47:52.463789 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463735    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0" containerName="payload-processing"
Nov 01 14:47:52.463789 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463752    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e392998c-8d13-4a07-9020-c9bf3292a2c4" containerName="maas-api"
Nov 01 14:47:52.463789 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463758    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e392998c-8d13-4a07-9020-c9bf3292a2c4" containerName="maas-api"
Nov 01 14:47:52.463789 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463764    5423 cpu_manager.go:414] "containerMap: removing container" podUID="09d974fc-18ae-41ee-8bac-5e22365f1cd6" containerName="payload-pre-processing"
Nov 01 14:47:52.463789 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463770    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d974fc-18ae-41ee-8bac-5e22365f1cd6" containerName="payload-pre-processing"
Nov 01 14:47:52.463943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463888    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcfc9cfb-394b-45c3-bdd3-d92ed9491ba0" containerName="payload-processing"
Nov 01 14:47:52.463943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463896    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e392998c-8d13-4a07-9020-c9bf3292a2c4" containerName="maas-api"
Nov 01 14:47:52.463943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.463903    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d974fc-18ae-41ee-8bac-5e22365f1cd6" containerName="payload-pre-processing"
Nov 01 14:47:52.466284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.466263    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.523052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.523015    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"]
Nov 01 14:47:52.564903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.564858    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-metrics-certs\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.565077 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.565005    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ss7c\" (UniqueName: \"kubernetes.io/projected/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-kube-api-access-2ss7c\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.565136 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.565076    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.565284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.565266    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.570924 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.570897    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7c335a-serving-cert\""
Nov 01 14:47:52.666576 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.666488    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-metrics-certs\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.666728 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.666593    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ss7c\" (UniqueName: \"kubernetes.io/projected/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-kube-api-access-2ss7c\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.666728 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.666669    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.666823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.666805    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.666956 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:47:52.666937    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-serving-cert: secret "maas-api-e2e-ipp-b-7c335a-serving-cert" not found
Nov 01 14:47:52.667021 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:47:52.667003    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls podName:f4e8547b-a373-4e35-95fa-fe3844ae2ff6 nodeName:}" failed. No retries permitted until 2026-11-01 14:47:53.166989585 +0000 UTC m=+1813.332474877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls") pod "maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c" (UID: "f4e8547b-a373-4e35-95fa-fe3844ae2ff6") : secret "maas-api-e2e-ipp-b-7c335a-serving-cert" not found
Nov 01 14:47:52.667284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.667263    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.668975 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.668954    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-metrics-certs\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.779547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.769090    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ss7c\" (UniqueName: \"kubernetes.io/projected/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-kube-api-access-2ss7c\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:52.907899 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.907847    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 14:47:52.908351 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:52.908315    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 14:47:53.179738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:53.179704    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:53.182293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:53.182265    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:53.379978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:53.379942    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:53.777284 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:47:53.777253    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e8547b_a373_4e35_95fa_fe3844ae2ff6.slice/crio-de57080eb3261d1b8ed67e5dc668ea60de84ba4ade2b85e54fbb18dc9fb87783 WatchSource:0}: Error finding container de57080eb3261d1b8ed67e5dc668ea60de84ba4ade2b85e54fbb18dc9fb87783: Status 404 returned error can't find the container with id de57080eb3261d1b8ed67e5dc668ea60de84ba4ade2b85e54fbb18dc9fb87783
Nov 01 14:47:53.780629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:53.780550    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"]
Nov 01 14:47:53.919582 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:53.919540    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 14:47:53.919784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:53.919613    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 14:47:54.469884 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:54.469848    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c" event={"ID":"f4e8547b-a373-4e35-95fa-fe3844ae2ff6","Type":"ContainerStarted","Data":"de57080eb3261d1b8ed67e5dc668ea60de84ba4ade2b85e54fbb18dc9fb87783"}
Nov 01 14:47:54.910332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:54.910303    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:47:54.911474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:54.911454    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:47:55.477992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:55.477959    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c" event={"ID":"f4e8547b-a373-4e35-95fa-fe3844ae2ff6","Type":"ContainerStarted","Data":"a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234"}
Nov 01 14:47:55.478461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:55.478170    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:47:55.595626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:47:55.595577    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c" podStartSLOduration=2.733141598 podStartE2EDuration="3.595562427s" totalImagesPullingTime="862.420829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:47:55.593967902 +0000 UTC m=+1815.759453215" watchObservedRunningTime="2026-11-01 14:47:55.595562427 +0000 UTC m=+1815.761047751"
Nov 01 14:48:01.490379 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:01.490349    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:48:02.468127 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:02.468083    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:04.326355 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:04.326307    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:05.150084 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:05.150040    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:48:06.565123 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:06.565084    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:06.565740 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:06.565713    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:06.567694 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:06.567662    5423 generic.go:356] "Generic (PLEG): container finished" podID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerID="8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972" exitCode=0
Nov 01 14:48:06.567819 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:06.567727    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" event={"ID":"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4","Type":"ContainerDied","Data":"8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972"}
Nov 01 14:48:06.568271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:06.568254    5423 scope.go:122] "RemoveContainer" containerID="8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972"
Nov 01 14:48:06.569882 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:06.569850    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:48:06.570000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:06.569922    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:48:06.570000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:06.569960    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:48:07.577727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:07.577689    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" event={"ID":"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4","Type":"ContainerStarted","Data":"7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771"}
Nov 01 14:48:08.315410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:08.313004    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:48:08.315410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:08.313171    5423 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:48:08.315410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:08.313231    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 14:48:08.558274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:08.558243    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:48:09.314712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:09.314675    5423 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:48:09.316101 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:09.315695    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 14:48:10.308283 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:10.308246    5423 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:48:10.308589 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:10.308556    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 14:48:11.313795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:11.313755    5423 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:48:11.314441 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:11.314406    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 14:48:12.319412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:12.317775    5423 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:48:12.319412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:12.317856    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 14:48:12.551346 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:12.551306    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:13.312352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:13.312321    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:48:13.313407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:13.313361    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:48:15.082944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:15.082902    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"]
Nov 01 14:48:15.190007 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:15.189979    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"]
Nov 01 14:48:19.307339 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:19.307299    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:20.083850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.083814    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2" podUID="bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34" containerName="payload-processing" containerID="cri-o://aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979" gracePeriod=25
Nov 01 14:48:20.222407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.202251    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk" podUID="e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a" containerName="payload-pre-processing" containerID="cri-o://634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d" gracePeriod=25
Nov 01 14:48:20.452637 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.452576    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:48:20.655162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.649747    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:48:20.663683 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.663648    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-plugins-config-volume\") pod \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\" (UID: \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\") "
Nov 01 14:48:20.663836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.663729    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-kube-api-access-zmt7l\" (UniqueName: \"kubernetes.io/projected/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-kube-api-access-zmt7l\") pod \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\" (UID: \"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34\") "
Nov 01 14:48:20.664712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.664693    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-plugins-config-volume" pod "bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34" (UID: "bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:20.673103 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.673071    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-kube-api-access-zmt7l" pod "bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34" (UID: "bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34"). InnerVolumeSpecName "kube-api-access-zmt7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:20.690525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.690501    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:48:20.706024 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.706000    5423 generic.go:356] "Generic (PLEG): container finished" podID="bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34" containerID="aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979" exitCode=0
Nov 01 14:48:20.706152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.706039    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"
Nov 01 14:48:20.706152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.706093    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2" event={"ID":"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34","Type":"ContainerDied","Data":"aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979"}
Nov 01 14:48:20.706276 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.706167    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2" event={"ID":"bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34","Type":"ContainerDied","Data":"d3e5817172701b82820d0cbc653c455f71628ed64737f27f83a1425e5ee0370b"}
Nov 01 14:48:20.706276 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.706201    5423 scope.go:122] "RemoveContainer" containerID="aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979"
Nov 01 14:48:20.707983 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.707957    5423 generic.go:356] "Generic (PLEG): container finished" podID="e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a" containerID="634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d" exitCode=0
Nov 01 14:48:20.708097 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.707987    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"
Nov 01 14:48:20.708097 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.708045    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk" event={"ID":"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a","Type":"ContainerDied","Data":"634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d"}
Nov 01 14:48:20.708216 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.708122    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk" event={"ID":"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a","Type":"ContainerDied","Data":"c1b01a27411902d612a806956912d12ff2fc531bf1fb3186e59eccedf12756d7"}
Nov 01 14:48:20.721188 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.721168    5423 scope.go:122] "RemoveContainer" containerID="aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979"
Nov 01 14:48:20.721525 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:20.721488    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979\": container with ID starting with aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979 not found: ID does not exist" containerID="aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979"
Nov 01 14:48:20.721622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.721531    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979"} err="failed to get container status \"aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979\": rpc error: code = NotFound desc = could not find container \"aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979\": container with ID starting with aa158b7fef5e9efcd0add9964c0c947803182c7a2611824f30d3c33f6f411979 not found: ID does not exist"
Nov 01 14:48:20.721622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.721549    5423 scope.go:122] "RemoveContainer" containerID="634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d"
Nov 01 14:48:20.730284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.730262    5423 scope.go:122] "RemoveContainer" containerID="634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d"
Nov 01 14:48:20.730567 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:20.730542    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d\": container with ID starting with 634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d not found: ID does not exist" containerID="634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d"
Nov 01 14:48:20.730645 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.730577    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d"} err="failed to get container status \"634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d\": rpc error: code = NotFound desc = could not find container \"634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d\": container with ID starting with 634c9bc9987ea5b70fcb9499cf8de1c278a9c9bc70b558cc80dabb1f63995e5d not found: ID does not exist"
Nov 01 14:48:20.764791 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.764748    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-plugins-config-volume\") pod \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\" (UID: \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\") "
Nov 01 14:48:20.765232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.764836    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-kube-api-access-mtz2m\" (UniqueName: \"kubernetes.io/projected/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-kube-api-access-mtz2m\") pod \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\" (UID: \"e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a\") "
Nov 01 14:48:20.765232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.765204    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-plugins-config-volume" pod "e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a" (UID: "e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:20.765636 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.765604    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:20.765636 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.765631    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:20.765798 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.765648    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmt7l\" (UniqueName: \"kubernetes.io/projected/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34-kube-api-access-zmt7l\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:20.767374 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.767349    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-kube-api-access-mtz2m" pod "e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a" (UID: "e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a"). InnerVolumeSpecName "kube-api-access-mtz2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:20.867250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.867215    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtz2m\" (UniqueName: \"kubernetes.io/projected/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a-kube-api-access-mtz2m\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:20.999889 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:20.999814    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"]
Nov 01 14:48:21.004577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:21.004550    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-9cb9f979-l6ll2"]
Nov 01 14:48:21.131126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:21.131096    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"]
Nov 01 14:48:21.134031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:21.134004    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-db98797cf-j7hxk"]
Nov 01 14:48:21.960897 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:21.960865    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34" path="/var/lib/kubelet/pods/bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34/volumes"
Nov 01 14:48:21.961353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:21.961337    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a" path="/var/lib/kubelet/pods/e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a/volumes"
Nov 01 14:48:22.625476 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:22.625445    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:26.539604 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:26.539572    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"]
Nov 01 14:48:26.540024 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:26.539787    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy" containerID="cri-o://74c7bff8abeda310c069eecc62c49fd86f3f9a307be1400a8bc78a63a1bb8dbc" gracePeriod=30
Nov 01 14:48:26.591590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:26.591558    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"]
Nov 01 14:48:26.591795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:26.591769    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy" containerID="cri-o://86a05188262021468e1ad4989ca890a44204611bd59cf058edca4df20e1f4a4c" gracePeriod=30
Nov 01 14:48:29.033325 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:29.033280    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:48:29.033738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:29.033350    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:48:29.320910 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:29.320876    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:48:29.321091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:29.320942    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:48:31.785147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.785108    5423 generic.go:356] "Generic (PLEG): container finished" podID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerID="74c7bff8abeda310c069eecc62c49fd86f3f9a307be1400a8bc78a63a1bb8dbc" exitCode=0
Nov 01 14:48:31.785631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.785244    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" event={"ID":"98d9712d-03fe-41d7-b79c-f5f0de92c8a1","Type":"ContainerDied","Data":"74c7bff8abeda310c069eecc62c49fd86f3f9a307be1400a8bc78a63a1bb8dbc"}
Nov 01 14:48:31.787621 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.787592    5423 generic.go:356] "Generic (PLEG): container finished" podID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerID="86a05188262021468e1ad4989ca890a44204611bd59cf058edca4df20e1f4a4c" exitCode=0
Nov 01 14:48:31.787748 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.787692    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" event={"ID":"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db","Type":"ContainerDied","Data":"86a05188262021468e1ad4989ca890a44204611bd59cf058edca4df20e1f4a4c"}
Nov 01 14:48:31.867697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.867668    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:48:31.901984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.901956    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:48:31.911108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911078    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-credential-socket\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911147    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-certs\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911315 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911299    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-envoy\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911339    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-kube-api-access-rhhmn\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911431 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911382    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istiod-ca-cert\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911412    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-credential-socket" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:31.911489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911470    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-socket\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911501    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-data\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911520    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-certs" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:31.911586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911573    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-token\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-token\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911627    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-podinfo\") pod \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\" (UID: \"98d9712d-03fe-41d7-b79c-f5f0de92c8a1\") "
Nov 01 14:48:31.911826 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911773    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-socket" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:31.911893 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911864    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-data" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:31.911949 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.911876    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istiod-ca-cert" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:31.912467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.912445    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:31.912467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.912466    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:31.912642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.912476    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:31.912642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.912491    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:31.912642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.912500    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:31.913658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.913634    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-kube-api-access-rhhmn" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "kube-api-access-rhhmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:31.913759 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.913698    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-token" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:31.913967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.913952    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-envoy" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:31.914031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:31.913979    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-podinfo" pod "98d9712d-03fe-41d7-b79c-f5f0de92c8a1" (UID: "98d9712d-03fe-41d7-b79c-f5f0de92c8a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:48:32.013173 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013141    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-socket\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013191    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-kube-api-access-jdkkc\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-kube-api-access-jdkkc\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013234    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-data\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013277    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istiod-ca-cert\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013326    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-certs\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013411    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-token\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-token\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013467    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-socket" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:32.013559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013483    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-envoy\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013553    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-data" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:32.013768 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013596    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-podinfo\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013768 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013657    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-credential-socket\") pod \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\" (UID: \"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db\") "
Nov 01 14:48:32.013768 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013694    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-certs" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:32.013925 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013781    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istiod-ca-cert" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:32.014009 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.013985    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-credential-socket" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:32.014295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014271    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.014409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014301    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.014409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014315    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.014409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014331    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-kube-api-access-rhhmn\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.014409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014347    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.014409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014360    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.014409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014372    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.014732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014416    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d9712d-03fe-41d7-b79c-f5f0de92c8a1-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.014732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.014431    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.015687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.015659    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-envoy" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:32.015932 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.015907    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-podinfo" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:48:32.015932 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.015911    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-kube-api-access-jdkkc" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "kube-api-access-jdkkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:32.016055 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.015997    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-token" pod "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" (UID: "55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:32.115923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.115844    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.115923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.115872    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.115923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.115886    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.115923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.115895    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdkkc\" (UniqueName: \"kubernetes.io/projected/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db-kube-api-access-jdkkc\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:32.699775 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:32.699738    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:32.795345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.795304    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq" event={"ID":"98d9712d-03fe-41d7-b79c-f5f0de92c8a1","Type":"ContainerDied","Data":"8a40ee809d515d310de06c58a75ee66309b68e23f6faaf5bff78702b051d1284"}
Nov 01 14:48:32.795843 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.795355    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"
Nov 01 14:48:32.795843 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.795433    5423 scope.go:122] "RemoveContainer" containerID="74c7bff8abeda310c069eecc62c49fd86f3f9a307be1400a8bc78a63a1bb8dbc"
Nov 01 14:48:32.797539 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.797507    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x" event={"ID":"55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db","Type":"ContainerDied","Data":"cecbf2fc1639dcc1d104e1c9437d6e0eb4d29a77323eebc6440a498604bfa5fd"}
Nov 01 14:48:32.797701 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.797625    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"
Nov 01 14:48:32.808706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.808690    5423 scope.go:122] "RemoveContainer" containerID="86a05188262021468e1ad4989ca890a44204611bd59cf058edca4df20e1f4a4c"
Nov 01 14:48:32.965811 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:32.965777    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"]
Nov 01 14:48:33.017073 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:33.017041    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-vhl5x"]
Nov 01 14:48:33.081191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:33.081102    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"]
Nov 01 14:48:33.085405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:33.085354    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-mrlgq"]
Nov 01 14:48:33.960780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:33.960736    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" path="/var/lib/kubelet/pods/55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db/volumes"
Nov 01 14:48:33.961446 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:33.961426    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" path="/var/lib/kubelet/pods/98d9712d-03fe-41d7-b79c-f5f0de92c8a1/volumes"
Nov 01 14:48:34.345243 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:34.345203    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-conmon-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfc9cfb_394b_45c3_bdd3_d92ed9491ba0.slice/crio-f28c1ac80eae43bef0fb2d4eb42f1e6b3cbbb293ef92e69c01fab6a8874a1f19.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:48:35.149207 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:35.149174    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:48:43.384878 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:43.384848    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"]
Nov 01 14:48:47.423286 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:47.423248    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"]
Nov 01 14:48:47.423704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:47.423465    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy" containerID="cri-o://e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50" gracePeriod=30
Nov 01 14:48:47.474953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:47.474905    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"]
Nov 01 14:48:47.475157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:47.475117    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" containerID="cri-o://d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec" gracePeriod=30
Nov 01 14:48:48.144353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.144319    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:48:48.144585 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.144402    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:48:48.385933 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.385880    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb" podUID="d594124e-2f15-4803-bcab-050ffddbea69" containerName="payload-pre-processing" containerID="cri-o://ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34" gracePeriod=25
Nov 01 14:48:48.648279 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.648251    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:48:48.745840 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.745767    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d594124e-2f15-4803-bcab-050ffddbea69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d594124e-2f15-4803-bcab-050ffddbea69-plugins-config-volume\") pod \"d594124e-2f15-4803-bcab-050ffddbea69\" (UID: \"d594124e-2f15-4803-bcab-050ffddbea69\") "
Nov 01 14:48:48.746017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.746001    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d594124e-2f15-4803-bcab-050ffddbea69-kube-api-access-spbh4\" (UniqueName: \"kubernetes.io/projected/d594124e-2f15-4803-bcab-050ffddbea69-kube-api-access-spbh4\") pod \"d594124e-2f15-4803-bcab-050ffddbea69\" (UID: \"d594124e-2f15-4803-bcab-050ffddbea69\") "
Nov 01 14:48:48.746113 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.746092    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d594124e-2f15-4803-bcab-050ffddbea69-plugins-config-volume" pod "d594124e-2f15-4803-bcab-050ffddbea69" (UID: "d594124e-2f15-4803-bcab-050ffddbea69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:48.746428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.746410    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d594124e-2f15-4803-bcab-050ffddbea69-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:48.747952 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.747932    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d594124e-2f15-4803-bcab-050ffddbea69-kube-api-access-spbh4" pod "d594124e-2f15-4803-bcab-050ffddbea69" (UID: "d594124e-2f15-4803-bcab-050ffddbea69"). InnerVolumeSpecName "kube-api-access-spbh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:48.848185 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.848146    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbh4\" (UniqueName: \"kubernetes.io/projected/d594124e-2f15-4803-bcab-050ffddbea69-kube-api-access-spbh4\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:48.949361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.949324    5423 generic.go:356] "Generic (PLEG): container finished" podID="d594124e-2f15-4803-bcab-050ffddbea69" containerID="ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34" exitCode=0
Nov 01 14:48:48.949559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.949371    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"
Nov 01 14:48:48.949559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.949381    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb" event={"ID":"d594124e-2f15-4803-bcab-050ffddbea69","Type":"ContainerDied","Data":"ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34"}
Nov 01 14:48:48.949559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.949455    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb" event={"ID":"d594124e-2f15-4803-bcab-050ffddbea69","Type":"ContainerDied","Data":"19d6fd5de720a92eb37a4c9e1f1333d6dc5e7973d035205adb285fa7b589e228"}
Nov 01 14:48:48.949559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.949479    5423 scope.go:122] "RemoveContainer" containerID="ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34"
Nov 01 14:48:48.959355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.959337    5423 scope.go:122] "RemoveContainer" containerID="ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34"
Nov 01 14:48:48.959662 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:48.959640    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34\": container with ID starting with ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34 not found: ID does not exist" containerID="ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34"
Nov 01 14:48:48.959735 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:48.959669    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34"} err="failed to get container status \"ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34\": rpc error: code = NotFound desc = could not find container \"ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34\": container with ID starting with ff83c690ca0a8c9a364a847f8b891415305f752f570f3dacdc930d32a7934e34 not found: ID does not exist"
Nov 01 14:48:49.119755 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:49.119716    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"]
Nov 01 14:48:49.124682 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:49.124654    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7789f6b699-lwwzb"]
Nov 01 14:48:49.416434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:49.416342    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:48:49.416434 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:49.416412    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:48:49.960044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:49.960007    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d594124e-2f15-4803-bcab-050ffddbea69" path="/var/lib/kubelet/pods/d594124e-2f15-4803-bcab-050ffddbea69/volumes"
Nov 01 14:48:52.735727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.735687    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:48:52.754353 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.754328    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:48:52.790498 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790473    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86908c75-744e-4256-9377-243eaf12e5c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86908c75-744e-4256-9377-243eaf12e5c2-istio-podinfo\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.790645 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790518    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-data\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.790645 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790555    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-envoy\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.790645 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790597    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-socket\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.790811 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790665    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-envoy\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.790811 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790698    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-data\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.790811 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790755    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-credential-socket\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.790811 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790804    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-token\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-token\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.791025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790833    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-podinfo\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.791025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790863    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-certs\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.791025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790854    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-data" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.791025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790972    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-certs\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.791235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791042    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-istio-token\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-istio-token\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.791235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791078    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-socket\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.791235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791121    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86908c75-744e-4256-9377-243eaf12e5c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86908c75-744e-4256-9377-243eaf12e5c2-istiod-ca-cert\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.791235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791176    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-kube-api-access-6gqfq\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-kube-api-access-6gqfq\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.791235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791218    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-kube-api-access-89llq\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-kube-api-access-89llq\") pod \"86908c75-744e-4256-9377-243eaf12e5c2\" (UID: \"86908c75-744e-4256-9377-243eaf12e5c2\") "
Nov 01 14:48:52.791534 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791262    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f950b9a7-0899-41ea-bea1-820bd450a92c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f950b9a7-0899-41ea-bea1-820bd450a92c-istiod-ca-cert\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.791534 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791318    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-credential-socket\") pod \"f950b9a7-0899-41ea-bea1-820bd450a92c\" (UID: \"f950b9a7-0899-41ea-bea1-820bd450a92c\") "
Nov 01 14:48:52.791534 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.790970    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-data" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.791700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791136    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-certs" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.791756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791149    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-certs" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.791756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791284    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-socket" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.791756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791498    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-credential-socket" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.791756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791721    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-credential-socket" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.791976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791953    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.792043 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791977    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.792043 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.791991    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.792043 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.792005    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.792043 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.792015    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f950b9a7-0899-41ea-bea1-820bd450a92c-istiod-ca-cert" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:52.792043 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.792019    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.792295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.792040    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86908c75-744e-4256-9377-243eaf12e5c2-istiod-ca-cert" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:52.792295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.792062    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.792295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.792072    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-socket" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.792295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.792081    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.793371 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.793345    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86908c75-744e-4256-9377-243eaf12e5c2-istio-podinfo" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:48:52.793510 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.793434    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-envoy" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.793581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.793519    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-envoy" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:48:52.793802 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.793769    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-istio-token" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:52.793913 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.793830    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-podinfo" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:48:52.794143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.794125    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-token" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:52.794903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.794887    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-kube-api-access-6gqfq" pod "f950b9a7-0899-41ea-bea1-820bd450a92c" (UID: "f950b9a7-0899-41ea-bea1-820bd450a92c"). InnerVolumeSpecName "kube-api-access-6gqfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:52.795224 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.795200    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-kube-api-access-89llq" pod "86908c75-744e-4256-9377-243eaf12e5c2" (UID: "86908c75-744e-4256-9377-243eaf12e5c2"). InnerVolumeSpecName "kube-api-access-89llq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:52.893263 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893229    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86908c75-744e-4256-9377-243eaf12e5c2-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893263 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893257    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893263 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893267    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86908c75-744e-4256-9377-243eaf12e5c2-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893275    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893283    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f950b9a7-0899-41ea-bea1-820bd450a92c-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893294    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893302    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f950b9a7-0899-41ea-bea1-820bd450a92c-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893310    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86908c75-744e-4256-9377-243eaf12e5c2-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893319    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqfq\" (UniqueName: \"kubernetes.io/projected/f950b9a7-0899-41ea-bea1-820bd450a92c-kube-api-access-6gqfq\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893329    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89llq\" (UniqueName: \"kubernetes.io/projected/86908c75-744e-4256-9377-243eaf12e5c2-kube-api-access-89llq\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.893531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.893337    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f950b9a7-0899-41ea-bea1-820bd450a92c-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:52.974756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.974674    5423 generic.go:356] "Generic (PLEG): container finished" podID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerID="e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50" exitCode=0
Nov 01 14:48:52.974756 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.974722    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"
Nov 01 14:48:52.974976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.974757    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" event={"ID":"f950b9a7-0899-41ea-bea1-820bd450a92c","Type":"ContainerDied","Data":"e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50"}
Nov 01 14:48:52.974976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.974814    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk" event={"ID":"f950b9a7-0899-41ea-bea1-820bd450a92c","Type":"ContainerDied","Data":"2ac02046a5ea9b7076a38d1a40e556b2a16539efb6102acf13e7417f5bae85d6"}
Nov 01 14:48:52.974976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.974845    5423 scope.go:122] "RemoveContainer" containerID="e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50"
Nov 01 14:48:52.976315 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.976289    5423 generic.go:356] "Generic (PLEG): container finished" podID="86908c75-744e-4256-9377-243eaf12e5c2" containerID="d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec" exitCode=0
Nov 01 14:48:52.976482 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.976331    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"
Nov 01 14:48:52.976536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.976486    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" event={"ID":"86908c75-744e-4256-9377-243eaf12e5c2","Type":"ContainerDied","Data":"d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec"}
Nov 01 14:48:52.976536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.976525    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn" event={"ID":"86908c75-744e-4256-9377-243eaf12e5c2","Type":"ContainerDied","Data":"0f74f43ecaab9d8dfc9196863d440c5b4f2a8fd54f1320bd5602d72ea748c23a"}
Nov 01 14:48:52.987279 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.987257    5423 scope.go:122] "RemoveContainer" containerID="e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50"
Nov 01 14:48:52.987660 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:52.987597    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50\": container with ID starting with e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50 not found: ID does not exist" containerID="e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50"
Nov 01 14:48:52.987660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.987628    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50"} err="failed to get container status \"e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50\": rpc error: code = NotFound desc = could not find container \"e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50\": container with ID starting with e9038eb70956de84436a69f4983b8fbe05957810ecf889ed108f8946d10c1c50 not found: ID does not exist"
Nov 01 14:48:52.987660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.987650    5423 scope.go:122] "RemoveContainer" containerID="d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec"
Nov 01 14:48:52.996313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.996297    5423 scope.go:122] "RemoveContainer" containerID="d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec"
Nov 01 14:48:52.996583 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:52.996562    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec\": container with ID starting with d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec not found: ID does not exist" containerID="d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec"
Nov 01 14:48:52.996640 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:52.996591    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec"} err="failed to get container status \"d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec\": rpc error: code = NotFound desc = could not find container \"d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec\": container with ID starting with d656364709a7cb60ab3252614471dd830856bb9020625ab614544fe016eae1ec not found: ID does not exist"
Nov 01 14:48:53.156131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:53.156072    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"]
Nov 01 14:48:53.157093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:53.157069    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-w4wtk"]
Nov 01 14:48:53.265279 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:53.265213    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"]
Nov 01 14:48:53.275514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:53.275488    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-75nwn"]
Nov 01 14:48:53.960030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:53.959991    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86908c75-744e-4256-9377-243eaf12e5c2" path="/var/lib/kubelet/pods/86908c75-744e-4256-9377-243eaf12e5c2/volumes"
Nov 01 14:48:53.960503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:53.960481    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" path="/var/lib/kubelet/pods/f950b9a7-0899-41ea-bea1-820bd450a92c/volumes"
Nov 01 14:48:57.439461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.436416    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"]
Nov 01 14:48:57.439461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.436685    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt" podUID="1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" containerName="maas-api" containerID="cri-o://835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a" gracePeriod=30
Nov 01 14:48:57.764894 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.764871    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:48:57.862456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.861269    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-kube-api-access-vxxms\" (UniqueName: \"kubernetes.io/projected/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-kube-api-access-vxxms\") pod \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") "
Nov 01 14:48:57.862456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.861320    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls\") pod \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") "
Nov 01 14:48:57.862456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.861445    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-metrics-certs\") pod \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") "
Nov 01 14:48:57.862456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.861591    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-openshift-service-ca\") pod \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\" (UID: \"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9\") "
Nov 01 14:48:57.863078 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.863006    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-openshift-service-ca" pod "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" (UID: "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:57.885952 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.885010    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-metrics-certs" pod "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" (UID: "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:48:57.891653 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.891475    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-kube-api-access-vxxms" pod "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" (UID: "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9"). InnerVolumeSpecName "kube-api-access-vxxms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:57.892029 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.892000    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls" pod "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" (UID: "1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:48:57.895803 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.895775    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"]
Nov 01 14:48:57.896044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.896016    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml" podUID="31f3bc07-4b36-42ba-8efb-2a03ebc4ce42" containerName="payload-processing" containerID="cri-o://38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed" gracePeriod=30
Nov 01 14:48:57.963059 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.963027    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxxms\" (UniqueName: \"kubernetes.io/projected/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-kube-api-access-vxxms\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:57.963206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.963061    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:57.963206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.963079    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:57.963206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:57.963093    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:58.009124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.009050    5423 generic.go:356] "Generic (PLEG): container finished" podID="1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" containerID="835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a" exitCode=0
Nov 01 14:48:58.009124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.009108    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"
Nov 01 14:48:58.009305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.009121    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt" event={"ID":"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9","Type":"ContainerDied","Data":"835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a"}
Nov 01 14:48:58.009305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.009173    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt" event={"ID":"1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9","Type":"ContainerDied","Data":"b5f07778811abb6c4411176acbe9dac20e0757b525df1976c6f24e778309bb50"}
Nov 01 14:48:58.009305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.009195    5423 scope.go:122] "RemoveContainer" containerID="835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a"
Nov 01 14:48:58.019435 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.019412    5423 scope.go:122] "RemoveContainer" containerID="835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a"
Nov 01 14:48:58.019732 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:58.019709    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a\": container with ID starting with 835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a not found: ID does not exist" containerID="835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a"
Nov 01 14:48:58.019861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.019741    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a"} err="failed to get container status \"835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a\": rpc error: code = NotFound desc = could not find container \"835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a\": container with ID starting with 835bdd02666a98424c1347be21f699f3a908e336f1e0f8c74d58a42b2b77ba8a not found: ID does not exist"
Nov 01 14:48:58.135754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.135720    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"]
Nov 01 14:48:58.139062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.139035    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbac83-85bbfffdf5-ms9xt"]
Nov 01 14:48:58.153901 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.153879    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:48:58.164967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.164945    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-plugins-config-volume\") pod \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\" (UID: \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\") "
Nov 01 14:48:58.165069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.165052    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-kube-api-access-tq8s9\" (UniqueName: \"kubernetes.io/projected/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-kube-api-access-tq8s9\") pod \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\" (UID: \"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42\") "
Nov 01 14:48:58.165357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.165336    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-plugins-config-volume" pod "31f3bc07-4b36-42ba-8efb-2a03ebc4ce42" (UID: "31f3bc07-4b36-42ba-8efb-2a03ebc4ce42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:58.165596 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.165576    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:58.167277 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.167247    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-kube-api-access-tq8s9" pod "31f3bc07-4b36-42ba-8efb-2a03ebc4ce42" (UID: "31f3bc07-4b36-42ba-8efb-2a03ebc4ce42"). InnerVolumeSpecName "kube-api-access-tq8s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:58.266626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:58.266547    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq8s9\" (UniqueName: \"kubernetes.io/projected/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42-kube-api-access-tq8s9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:59.017861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.017828    5423 generic.go:356] "Generic (PLEG): container finished" podID="31f3bc07-4b36-42ba-8efb-2a03ebc4ce42" containerID="38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed" exitCode=0
Nov 01 14:48:59.018367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.017893    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"
Nov 01 14:48:59.018367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.017910    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml" event={"ID":"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42","Type":"ContainerDied","Data":"38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed"}
Nov 01 14:48:59.018367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.017971    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml" event={"ID":"31f3bc07-4b36-42ba-8efb-2a03ebc4ce42","Type":"ContainerDied","Data":"c0fe9c8159e2de10fcb1c00f1b7b3d384b0a8c6fbb019fd4b631f279b051d4d2"}
Nov 01 14:48:59.018367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.017995    5423 scope.go:122] "RemoveContainer" containerID="38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed"
Nov 01 14:48:59.037656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.037632    5423 scope.go:122] "RemoveContainer" containerID="38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed"
Nov 01 14:48:59.037957 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:48:59.037936    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed\": container with ID starting with 38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed not found: ID does not exist" containerID="38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed"
Nov 01 14:48:59.038030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.037970    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed"} err="failed to get container status \"38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed\": rpc error: code = NotFound desc = could not find container \"38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed\": container with ID starting with 38abd9518b6f3863d47ff81a747440ef5af5c2a917d0dcdf1f09da8c20ba59ed not found: ID does not exist"
Nov 01 14:48:59.201612 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.201576    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"]
Nov 01 14:48:59.204707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.204682    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-5f9fdd6cc6-d8dml"]
Nov 01 14:48:59.964904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.964866    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" path="/var/lib/kubelet/pods/1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9/volumes"
Nov 01 14:48:59.965349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:48:59.965327    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f3bc07-4b36-42ba-8efb-2a03ebc4ce42" path="/var/lib/kubelet/pods/31f3bc07-4b36-42ba-8efb-2a03ebc4ce42/volumes"
Nov 01 14:49:02.000157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:02.000118    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"]
Nov 01 14:49:03.357192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357157    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"]
Nov 01 14:49:03.357771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357753    5423 cpu_manager.go:414] "containerMap: removing container" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357772    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357791    5423 cpu_manager.go:414] "containerMap: removing container" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357798    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357813    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" containerName="maas-api"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357818    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" containerName="maas-api"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357831    5423 cpu_manager.go:414] "containerMap: removing container" podUID="31f3bc07-4b36-42ba-8efb-2a03ebc4ce42" containerName="payload-processing"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357836    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f3bc07-4b36-42ba-8efb-2a03ebc4ce42" containerName="payload-processing"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357855    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy"
Nov 01 14:49:03.357859 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357860    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357874    5423 cpu_manager.go:414] "containerMap: removing container" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357880    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357890    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d594124e-2f15-4803-bcab-050ffddbea69" containerName="payload-pre-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357895    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d594124e-2f15-4803-bcab-050ffddbea69" containerName="payload-pre-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357903    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a" containerName="payload-pre-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357908    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a" containerName="payload-pre-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357923    5423 cpu_manager.go:414] "containerMap: removing container" podUID="bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34" containerName="payload-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.357930    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34" containerName="payload-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358019    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f950b9a7-0899-41ea-bea1-820bd450a92c" containerName="istio-proxy"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358031    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="86908c75-744e-4256-9377-243eaf12e5c2" containerName="istio-proxy"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358044    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d594124e-2f15-4803-bcab-050ffddbea69" containerName="payload-pre-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358051    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ef7c29-67fd-4e8b-b799-8f23ba3cc78a" containerName="payload-pre-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358058    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6ed4d2-ac42-44e3-b3ab-b74ecc8dba34" containerName="payload-processing"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358064    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d9712d-03fe-41d7-b79c-f5f0de92c8a1" containerName="istio-proxy"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358071    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e3aa7b-02db-4b4d-aa3f-d9c74d47f4db" containerName="istio-proxy"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358079    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfac50d-0c1a-4bdb-bf9e-4c7955df34b9" containerName="maas-api"
Nov 01 14:49:03.358205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.358089    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f3bc07-4b36-42ba-8efb-2a03ebc4ce42" containerName="payload-processing"
Nov 01 14:49:03.361451 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.361432    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.415784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.415757    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d195c1-openshift-default-dockercfg-6wfh9\""
Nov 01 14:49:03.469563 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.469532    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"]
Nov 01 14:49:03.490464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490433    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-socket\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.490464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490466    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-envoy\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.490649 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490501    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-credential-socket\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.490649 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490574    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x647j\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-kube-api-access-x647j\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.490727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490667    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-podinfo\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.490727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490696    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec6d420-5b16-4a0b-868c-819eea8a7368-istiod-ca-cert\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.490844 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490782    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-certs\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.490844 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490820    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-token\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.491016 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.490986    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-data\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.594796 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.592563    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-certs\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.594796 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.593372    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-certs\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.594796 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.592619    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-token\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.595132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.595106    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-data\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.595192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.595154    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-socket\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.595192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.595180    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-envoy\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.595287 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.595229    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-credential-socket\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.595287 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.595265    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x647j\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-kube-api-access-x647j\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.595368 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.595338    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-podinfo\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.595436 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.595379    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec6d420-5b16-4a0b-868c-819eea8a7368-istiod-ca-cert\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.596118 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.596093    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec6d420-5b16-4a0b-868c-819eea8a7368-istiod-ca-cert\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.596344 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.596321    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-data\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.596591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.596554    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-socket\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.597271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.597215    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-credential-socket\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.599088 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.599063    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-envoy\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.599310 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.599293    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-podinfo\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.685108 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.684982    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-token\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.712178 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.712109    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x647j\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-kube-api-access-x647j\") pod \"e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") " pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:03.974635 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:03.974559    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:04.390055 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:49:04.390023    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec6d420_5b16_4a0b_868c_819eea8a7368.slice/crio-a9e6356df948a3cdc1ce3f5bf0e32339151c6a728ef02f25c536c409edfa9440 WatchSource:0}: Error finding container a9e6356df948a3cdc1ce3f5bf0e32339151c6a728ef02f25c536c409edfa9440: Status 404 returned error can't find the container with id a9e6356df948a3cdc1ce3f5bf0e32339151c6a728ef02f25c536c409edfa9440
Nov 01 14:49:04.392717 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:04.392680    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:49:04.392795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:04.392767    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:49:04.392833 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:04.392807    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:49:04.393771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:04.393752    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"]
Nov 01 14:49:05.071800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:05.071763    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" event={"ID":"8ec6d420-5b16-4a0b-868c-819eea8a7368","Type":"ContainerStarted","Data":"a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f"}
Nov 01 14:49:05.072014 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:05.071820    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" event={"ID":"8ec6d420-5b16-4a0b-868c-819eea8a7368","Type":"ContainerStarted","Data":"a9e6356df948a3cdc1ce3f5bf0e32339151c6a728ef02f25c536c409edfa9440"}
Nov 01 14:49:05.149799 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:49:05.149762    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:49:05.243918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:05.243863    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" podStartSLOduration=2.243844815 podStartE2EDuration="2.243844815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:05.240549968 +0000 UTC m=+1885.406035266" watchObservedRunningTime="2026-11-01 14:49:05.243844815 +0000 UTC m=+1885.409330116"
Nov 01 14:49:05.975592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:05.975559    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:05.981223 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:05.981197    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:06.077308 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:06.077270    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:06.078147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:06.078130    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:49:07.001189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:07.001131    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs" podUID="405734c6-9251-48e5-ae92-8fad2d3c1c64" containerName="payload-processing" containerID="cri-o://20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c" gracePeriod=25
Nov 01 14:49:07.266904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:07.266844    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:49:07.357156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:07.357126    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405734c6-9251-48e5-ae92-8fad2d3c1c64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405734c6-9251-48e5-ae92-8fad2d3c1c64-plugins-config-volume\") pod \"405734c6-9251-48e5-ae92-8fad2d3c1c64\" (UID: \"405734c6-9251-48e5-ae92-8fad2d3c1c64\") "
Nov 01 14:49:07.357341 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:07.357200    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405734c6-9251-48e5-ae92-8fad2d3c1c64-kube-api-access-rhhww\" (UniqueName: \"kubernetes.io/projected/405734c6-9251-48e5-ae92-8fad2d3c1c64-kube-api-access-rhhww\") pod \"405734c6-9251-48e5-ae92-8fad2d3c1c64\" (UID: \"405734c6-9251-48e5-ae92-8fad2d3c1c64\") "
Nov 01 14:49:07.357580 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:07.357557    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405734c6-9251-48e5-ae92-8fad2d3c1c64-plugins-config-volume" pod "405734c6-9251-48e5-ae92-8fad2d3c1c64" (UID: "405734c6-9251-48e5-ae92-8fad2d3c1c64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:49:07.359281 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:07.359259    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405734c6-9251-48e5-ae92-8fad2d3c1c64-kube-api-access-rhhww" pod "405734c6-9251-48e5-ae92-8fad2d3c1c64" (UID: "405734c6-9251-48e5-ae92-8fad2d3c1c64"). InnerVolumeSpecName "kube-api-access-rhhww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:49:07.458329 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:07.458298    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405734c6-9251-48e5-ae92-8fad2d3c1c64-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:07.458329 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:07.458326    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhww\" (UniqueName: \"kubernetes.io/projected/405734c6-9251-48e5-ae92-8fad2d3c1c64-kube-api-access-rhhww\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:08.090396 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.090359    5423 generic.go:356] "Generic (PLEG): container finished" podID="405734c6-9251-48e5-ae92-8fad2d3c1c64" containerID="20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c" exitCode=0
Nov 01 14:49:08.090788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.090423    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"
Nov 01 14:49:08.090788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.090452    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs" event={"ID":"405734c6-9251-48e5-ae92-8fad2d3c1c64","Type":"ContainerDied","Data":"20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c"}
Nov 01 14:49:08.090788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.090511    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs" event={"ID":"405734c6-9251-48e5-ae92-8fad2d3c1c64","Type":"ContainerDied","Data":"0ca9d4fa26a9ac0691762b2ebfe5a9c4b6338db5622a5ea6d7fc0866216bd87a"}
Nov 01 14:49:08.090788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.090538    5423 scope.go:122] "RemoveContainer" containerID="20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c"
Nov 01 14:49:08.100033 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.100013    5423 scope.go:122] "RemoveContainer" containerID="20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c"
Nov 01 14:49:08.100306 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:49:08.100286    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c\": container with ID starting with 20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c not found: ID does not exist" containerID="20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c"
Nov 01 14:49:08.100355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.100312    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c"} err="failed to get container status \"20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c\": rpc error: code = NotFound desc = could not find container \"20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c\": container with ID starting with 20cc63a797e48aed8c3bfcdf1f55aed4c7bba5273de780ce0aabf8ab76d0594c not found: ID does not exist"
Nov 01 14:49:08.268027 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.267941    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"]
Nov 01 14:49:08.315436 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:08.315406    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-5f89cb494c-l9kjs"]
Nov 01 14:49:09.539916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.539882    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"]
Nov 01 14:49:09.540472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.540457    5423 cpu_manager.go:414] "containerMap: removing container" podUID="405734c6-9251-48e5-ae92-8fad2d3c1c64" containerName="payload-processing"
Nov 01 14:49:09.540543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.540474    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405734c6-9251-48e5-ae92-8fad2d3c1c64" containerName="payload-processing"
Nov 01 14:49:09.540587 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.540574    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="405734c6-9251-48e5-ae92-8fad2d3c1c64" containerName="payload-processing"
Nov 01 14:49:09.545239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.545217    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:09.597492 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.597466    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"]
Nov 01 14:49:09.645720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.645692    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-be7e2e\""
Nov 01 14:49:09.645860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.645810    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-be7e2e-dockercfg-2cqjd\""
Nov 01 14:49:09.691130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.691100    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqjr\" (UniqueName: \"kubernetes.io/projected/7130638f-3a21-4c18-9df9-23260b113eb3-kube-api-access-dlqjr\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq\" (UID: \"7130638f-3a21-4c18-9df9-23260b113eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:09.691264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.691153    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7130638f-3a21-4c18-9df9-23260b113eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq\" (UID: \"7130638f-3a21-4c18-9df9-23260b113eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:09.792904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.792818    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqjr\" (UniqueName: \"kubernetes.io/projected/7130638f-3a21-4c18-9df9-23260b113eb3-kube-api-access-dlqjr\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq\" (UID: \"7130638f-3a21-4c18-9df9-23260b113eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:09.792904 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.792895    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7130638f-3a21-4c18-9df9-23260b113eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq\" (UID: \"7130638f-3a21-4c18-9df9-23260b113eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:09.793654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.793635    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7130638f-3a21-4c18-9df9-23260b113eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq\" (UID: \"7130638f-3a21-4c18-9df9-23260b113eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:09.850971 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.850934    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqjr\" (UniqueName: \"kubernetes.io/projected/7130638f-3a21-4c18-9df9-23260b113eb3-kube-api-access-dlqjr\") pod \"payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq\" (UID: \"7130638f-3a21-4c18-9df9-23260b113eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:09.858822 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.858791    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:09.962898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:09.962847    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405734c6-9251-48e5-ae92-8fad2d3c1c64" path="/var/lib/kubelet/pods/405734c6-9251-48e5-ae92-8fad2d3c1c64/volumes"
Nov 01 14:49:10.232885 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:49:10.232852    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7130638f_3a21_4c18_9df9_23260b113eb3.slice/crio-93b797ee0b99fe8e7cb2448457bc99050d256f08107440572d4b8817488f3eed WatchSource:0}: Error finding container 93b797ee0b99fe8e7cb2448457bc99050d256f08107440572d4b8817488f3eed: Status 404 returned error can't find the container with id 93b797ee0b99fe8e7cb2448457bc99050d256f08107440572d4b8817488f3eed
Nov 01 14:49:10.236347 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.236329    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"]
Nov 01 14:49:10.358880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.358841    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"]
Nov 01 14:49:10.362807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.362782    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:10.417675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.417641    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"]
Nov 01 14:49:10.503269 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.503189    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxh59\" (UniqueName: \"kubernetes.io/projected/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-kube-api-access-fxh59\") pod \"payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg\" (UID: \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:10.503469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.503435    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg\" (UID: \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:10.604590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.604549    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg\" (UID: \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:10.605040 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.604642    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxh59\" (UniqueName: \"kubernetes.io/projected/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-kube-api-access-fxh59\") pod \"payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg\" (UID: \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:10.605290 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.605270    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg\" (UID: \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:10.663312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.663286    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxh59\" (UniqueName: \"kubernetes.io/projected/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-kube-api-access-fxh59\") pod \"payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg\" (UID: \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:10.680147 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:10.680127    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:11.048665 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:49:11.048634    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f773d43_c6b9_4da9_bf6e_0ea67c64ad7e.slice/crio-4ddbd84917f78794595e90ab90c1c22f04d526c7126f9262a3a0db8e00072cdc WatchSource:0}: Error finding container 4ddbd84917f78794595e90ab90c1c22f04d526c7126f9262a3a0db8e00072cdc: Status 404 returned error can't find the container with id 4ddbd84917f78794595e90ab90c1c22f04d526c7126f9262a3a0db8e00072cdc
Nov 01 14:49:11.051983 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.051957    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"]
Nov 01 14:49:11.172421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.169860    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg" event={"ID":"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e","Type":"ContainerStarted","Data":"4ddbd84917f78794595e90ab90c1c22f04d526c7126f9262a3a0db8e00072cdc"}
Nov 01 14:49:11.172421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.171737    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq" event={"ID":"7130638f-3a21-4c18-9df9-23260b113eb3","Type":"ContainerStarted","Data":"b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded"}
Nov 01 14:49:11.172421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.171774    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq" event={"ID":"7130638f-3a21-4c18-9df9-23260b113eb3","Type":"ContainerStarted","Data":"93b797ee0b99fe8e7cb2448457bc99050d256f08107440572d4b8817488f3eed"}
Nov 01 14:49:11.172421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.172210    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:11.375675 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.375623    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq" podStartSLOduration=2.375607355 podStartE2EDuration="2.375607355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:11.292348791 +0000 UTC m=+1891.457834090" watchObservedRunningTime="2026-11-01 14:49:11.375607355 +0000 UTC m=+1891.541092653"
Nov 01 14:49:11.378036 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.378008    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"]
Nov 01 14:49:11.382218 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.382198    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:11.435152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.435103    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"]
Nov 01 14:49:11.435517 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.435490    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d195c1\""
Nov 01 14:49:11.435639 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.435572    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d195c1-dockercfg-8qw5x\""
Nov 01 14:49:11.577259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.564635    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzz5z\" (UniqueName: \"kubernetes.io/projected/59d7fca4-25b7-4e2c-a04e-f8553a325b67-kube-api-access-mzz5z\") pod \"payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx\" (UID: \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:11.577259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.564836    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d7fca4-25b7-4e2c-a04e-f8553a325b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx\" (UID: \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:11.583865 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.583713    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"]
Nov 01 14:49:11.598030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.595890    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:11.616496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.616466    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"]
Nov 01 14:49:11.665959 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.665867    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d7fca4-25b7-4e2c-a04e-f8553a325b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx\" (UID: \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:11.666120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.665989    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzz5z\" (UniqueName: \"kubernetes.io/projected/59d7fca4-25b7-4e2c-a04e-f8553a325b67-kube-api-access-mzz5z\") pod \"payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx\" (UID: \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:11.666120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.666013    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jccs\" (UniqueName: \"kubernetes.io/projected/ba341519-804c-48d1-aa1c-be401139bbaf-kube-api-access-2jccs\") pod \"payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k\" (UID: \"ba341519-804c-48d1-aa1c-be401139bbaf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:11.666120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.666041    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba341519-804c-48d1-aa1c-be401139bbaf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k\" (UID: \"ba341519-804c-48d1-aa1c-be401139bbaf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:11.666591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.666571    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d7fca4-25b7-4e2c-a04e-f8553a325b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx\" (UID: \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:11.725064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.725029    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzz5z\" (UniqueName: \"kubernetes.io/projected/59d7fca4-25b7-4e2c-a04e-f8553a325b67-kube-api-access-mzz5z\") pod \"payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx\" (UID: \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:11.767657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.767624    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jccs\" (UniqueName: \"kubernetes.io/projected/ba341519-804c-48d1-aa1c-be401139bbaf-kube-api-access-2jccs\") pod \"payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k\" (UID: \"ba341519-804c-48d1-aa1c-be401139bbaf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:11.767657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.767667    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba341519-804c-48d1-aa1c-be401139bbaf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k\" (UID: \"ba341519-804c-48d1-aa1c-be401139bbaf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:11.768354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.768335    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba341519-804c-48d1-aa1c-be401139bbaf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k\" (UID: \"ba341519-804c-48d1-aa1c-be401139bbaf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:11.825596 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.825555    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jccs\" (UniqueName: \"kubernetes.io/projected/ba341519-804c-48d1-aa1c-be401139bbaf-kube-api-access-2jccs\") pod \"payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k\" (UID: \"ba341519-804c-48d1-aa1c-be401139bbaf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:11.916723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.916646    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:11.996734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:11.996607    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:12.180591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:12.180509    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg" event={"ID":"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e","Type":"ContainerStarted","Data":"291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92"}
Nov 01 14:49:12.180591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:12.180585    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:12.289009 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:49:12.288979    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba341519_804c_48d1_aa1c_be401139bbaf.slice/crio-266fb0805c6344657b6d88cae0846504619e9aaa93d74b350a819c733167e441 WatchSource:0}: Error finding container 266fb0805c6344657b6d88cae0846504619e9aaa93d74b350a819c733167e441: Status 404 returned error can't find the container with id 266fb0805c6344657b6d88cae0846504619e9aaa93d74b350a819c733167e441
Nov 01 14:49:12.291835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:12.291807    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"]
Nov 01 14:49:12.298004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:12.297949    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg" podStartSLOduration=2.297932137 podStartE2EDuration="2.297932137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:12.296715518 +0000 UTC m=+1892.462200819" watchObservedRunningTime="2026-11-01 14:49:12.297932137 +0000 UTC m=+1892.463417436"
Nov 01 14:49:12.393648 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:49:12.393621    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d7fca4_25b7_4e2c_a04e_f8553a325b67.slice/crio-b747c1c1bdcd3730086643dadb47648b4b3b96b1c9f2812d52f1d0a2375aad3a WatchSource:0}: Error finding container b747c1c1bdcd3730086643dadb47648b4b3b96b1c9f2812d52f1d0a2375aad3a: Status 404 returned error can't find the container with id b747c1c1bdcd3730086643dadb47648b4b3b96b1c9f2812d52f1d0a2375aad3a
Nov 01 14:49:12.398893 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:12.398867    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"]
Nov 01 14:49:13.189044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:13.189008    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx" event={"ID":"59d7fca4-25b7-4e2c-a04e-f8553a325b67","Type":"ContainerStarted","Data":"7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da"}
Nov 01 14:49:13.189522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:13.189065    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx" event={"ID":"59d7fca4-25b7-4e2c-a04e-f8553a325b67","Type":"ContainerStarted","Data":"b747c1c1bdcd3730086643dadb47648b4b3b96b1c9f2812d52f1d0a2375aad3a"}
Nov 01 14:49:13.189522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:13.189133    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:13.190701 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:13.190671    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k" event={"ID":"ba341519-804c-48d1-aa1c-be401139bbaf","Type":"ContainerStarted","Data":"d2fa78e298e3e6ee9870889a67a5fc8b087a83a592f2de36ff87bb0502677cb5"}
Nov 01 14:49:13.190827 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:13.190709    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k" event={"ID":"ba341519-804c-48d1-aa1c-be401139bbaf","Type":"ContainerStarted","Data":"266fb0805c6344657b6d88cae0846504619e9aaa93d74b350a819c733167e441"}
Nov 01 14:49:13.190827 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:13.190758    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:13.306579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:13.306526    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx" podStartSLOduration=2.306513609 podStartE2EDuration="2.306513609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:13.305369937 +0000 UTC m=+1893.470855235" watchObservedRunningTime="2026-11-01 14:49:13.306513609 +0000 UTC m=+1893.471998907"
Nov 01 14:49:17.754908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:17.754855    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k" podStartSLOduration=6.754841324 podStartE2EDuration="6.754841324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:13.438292041 +0000 UTC m=+1893.603777340" watchObservedRunningTime="2026-11-01 14:49:17.754841324 +0000 UTC m=+1897.920326621"
Nov 01 14:49:17.756640 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:17.756618    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:49:22.183622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:22.183542    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:49:23.193261 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:23.193231    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:49:24.200887 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:24.200857    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:49:24.201274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:24.200932    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:49:35.150232 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:49:35.150197    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:49:35.468741 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.468657    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"]
Nov 01 14:49:35.473271 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.473248    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:35.528914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.528890    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-34d384\""
Nov 01 14:49:35.572782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.572747    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcpz\" (UniqueName: \"kubernetes.io/projected/9605f980-140d-40a6-aaa5-45c36e8a05e9-kube-api-access-6tcpz\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc\" (UID: \"9605f980-140d-40a6-aaa5-45c36e8a05e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:35.572927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.572862    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9605f980-140d-40a6-aaa5-45c36e8a05e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc\" (UID: \"9605f980-140d-40a6-aaa5-45c36e8a05e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:35.579919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.579891    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-34d384-dockercfg-tlnwq\""
Nov 01 14:49:35.582742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.582721    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"]
Nov 01 14:49:35.673542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.673500    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcpz\" (UniqueName: \"kubernetes.io/projected/9605f980-140d-40a6-aaa5-45c36e8a05e9-kube-api-access-6tcpz\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc\" (UID: \"9605f980-140d-40a6-aaa5-45c36e8a05e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:35.673706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.673574    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9605f980-140d-40a6-aaa5-45c36e8a05e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc\" (UID: \"9605f980-140d-40a6-aaa5-45c36e8a05e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:35.674298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.674277    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9605f980-140d-40a6-aaa5-45c36e8a05e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc\" (UID: \"9605f980-140d-40a6-aaa5-45c36e8a05e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:35.731948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.731873    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcpz\" (UniqueName: \"kubernetes.io/projected/9605f980-140d-40a6-aaa5-45c36e8a05e9-kube-api-access-6tcpz\") pod \"payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc\" (UID: \"9605f980-140d-40a6-aaa5-45c36e8a05e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:35.787470 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:35.787442    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:36.153619 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:49:36.153583    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9605f980_140d_40a6_aaa5_45c36e8a05e9.slice/crio-badb40fb5e6e7cfe51e160fd6b1d4a291bb32ea1524ced9feb7ef1930568ddf5 WatchSource:0}: Error finding container badb40fb5e6e7cfe51e160fd6b1d4a291bb32ea1524ced9feb7ef1930568ddf5: Status 404 returned error can't find the container with id badb40fb5e6e7cfe51e160fd6b1d4a291bb32ea1524ced9feb7ef1930568ddf5
Nov 01 14:49:36.156302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.156279    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"]
Nov 01 14:49:36.279469 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.279441    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"]
Nov 01 14:49:36.283360 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.283336    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:36.333785 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.333747    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc" event={"ID":"9605f980-140d-40a6-aaa5-45c36e8a05e9","Type":"ContainerStarted","Data":"a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af"}
Nov 01 14:49:36.333967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.333820    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:36.333967 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.333844    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc" event={"ID":"9605f980-140d-40a6-aaa5-45c36e8a05e9","Type":"ContainerStarted","Data":"badb40fb5e6e7cfe51e160fd6b1d4a291bb32ea1524ced9feb7ef1930568ddf5"}
Nov 01 14:49:36.337521 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.337493    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"]
Nov 01 14:49:36.382573 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.382543    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0397dc-1006-4eff-a02b-9635a774f80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p\" (UID: \"7b0397dc-1006-4eff-a02b-9635a774f80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:36.382704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.382671    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8mwn\" (UniqueName: \"kubernetes.io/projected/7b0397dc-1006-4eff-a02b-9635a774f80a-kube-api-access-c8mwn\") pod \"payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p\" (UID: \"7b0397dc-1006-4eff-a02b-9635a774f80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:36.483772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.483689    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0397dc-1006-4eff-a02b-9635a774f80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p\" (UID: \"7b0397dc-1006-4eff-a02b-9635a774f80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:36.483916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.483780    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8mwn\" (UniqueName: \"kubernetes.io/projected/7b0397dc-1006-4eff-a02b-9635a774f80a-kube-api-access-c8mwn\") pod \"payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p\" (UID: \"7b0397dc-1006-4eff-a02b-9635a774f80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:36.484510 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.484490    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0397dc-1006-4eff-a02b-9635a774f80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p\" (UID: \"7b0397dc-1006-4eff-a02b-9635a774f80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:36.515550 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.515499    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc" podStartSLOduration=1.515484275 podStartE2EDuration="1.515484275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:36.51277601 +0000 UTC m=+1916.678261308" watchObservedRunningTime="2026-11-01 14:49:36.515484275 +0000 UTC m=+1916.680969572"
Nov 01 14:49:36.542323 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.542293    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8mwn\" (UniqueName: \"kubernetes.io/projected/7b0397dc-1006-4eff-a02b-9635a774f80a-kube-api-access-c8mwn\") pod \"payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p\" (UID: \"7b0397dc-1006-4eff-a02b-9635a774f80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:36.596782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:36.596740    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:37.013743 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:37.013704    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"]
Nov 01 14:49:37.343631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:37.343597    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p" event={"ID":"7b0397dc-1006-4eff-a02b-9635a774f80a","Type":"ContainerStarted","Data":"3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82"}
Nov 01 14:49:37.344113 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:37.343649    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p" event={"ID":"7b0397dc-1006-4eff-a02b-9635a774f80a","Type":"ContainerStarted","Data":"39c607358f25c46f605b60c4a5da94ca5f0b51326e8fd5d2aa72439319083029"}
Nov 01 14:49:37.344113 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:37.343913    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:37.462209 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:37.462151    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p" podStartSLOduration=1.4621326350000001 podStartE2EDuration="1.462132635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:37.459996737 +0000 UTC m=+1917.625482031" watchObservedRunningTime="2026-11-01 14:49:37.462132635 +0000 UTC m=+1917.627617933"
Nov 01 14:49:45.201503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.201470    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"]
Nov 01 14:49:45.201984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.201652    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x" podUID="9b391c51-7998-423f-bc7d-412cfd17f2a0" containerName="maas-api" containerID="cri-o://dd421d26859f6c6ca58f16b2d5a0b3b513d149fbe93e8665c52b5b44b29198a8" gracePeriod=30
Nov 01 14:49:45.401126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.401095    5423 generic.go:356] "Generic (PLEG): container finished" podID="9b391c51-7998-423f-bc7d-412cfd17f2a0" containerID="dd421d26859f6c6ca58f16b2d5a0b3b513d149fbe93e8665c52b5b44b29198a8" exitCode=0
Nov 01 14:49:45.401292 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.401143    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x" event={"ID":"9b391c51-7998-423f-bc7d-412cfd17f2a0","Type":"ContainerDied","Data":"dd421d26859f6c6ca58f16b2d5a0b3b513d149fbe93e8665c52b5b44b29198a8"}
Nov 01 14:49:45.501886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.501862    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:49:45.605247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.605205    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b391c51-7998-423f-bc7d-412cfd17f2a0-kube-api-access-ghxpv\" (UniqueName: \"kubernetes.io/projected/9b391c51-7998-423f-bc7d-412cfd17f2a0-kube-api-access-ghxpv\") pod \"9b391c51-7998-423f-bc7d-412cfd17f2a0\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") "
Nov 01 14:49:45.606070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.606046    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls\") pod \"9b391c51-7998-423f-bc7d-412cfd17f2a0\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") "
Nov 01 14:49:45.606657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.606635    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-metrics-certs\") pod \"9b391c51-7998-423f-bc7d-412cfd17f2a0\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") "
Nov 01 14:49:45.606901 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.606882    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b391c51-7998-423f-bc7d-412cfd17f2a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b391c51-7998-423f-bc7d-412cfd17f2a0-openshift-service-ca\") pod \"9b391c51-7998-423f-bc7d-412cfd17f2a0\" (UID: \"9b391c51-7998-423f-bc7d-412cfd17f2a0\") "
Nov 01 14:49:45.608048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.608024    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b391c51-7998-423f-bc7d-412cfd17f2a0-openshift-service-ca" pod "9b391c51-7998-423f-bc7d-412cfd17f2a0" (UID: "9b391c51-7998-423f-bc7d-412cfd17f2a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:49:45.643669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.642350    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls" pod "9b391c51-7998-423f-bc7d-412cfd17f2a0" (UID: "9b391c51-7998-423f-bc7d-412cfd17f2a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:49:45.652240 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.652190    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b391c51-7998-423f-bc7d-412cfd17f2a0-kube-api-access-ghxpv" pod "9b391c51-7998-423f-bc7d-412cfd17f2a0" (UID: "9b391c51-7998-423f-bc7d-412cfd17f2a0"). InnerVolumeSpecName "kube-api-access-ghxpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:49:45.658562 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.658511    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-metrics-certs" pod "9b391c51-7998-423f-bc7d-412cfd17f2a0" (UID: "9b391c51-7998-423f-bc7d-412cfd17f2a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:49:45.669764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.669130    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"]
Nov 01 14:49:45.669764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.669424    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s" podUID="65a92e04-f8d0-4bc8-9fcd-33dd819603f0" containerName="payload-processing" containerID="cri-o://de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81" gracePeriod=30
Nov 01 14:49:45.709002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.708973    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghxpv\" (UniqueName: \"kubernetes.io/projected/9b391c51-7998-423f-bc7d-412cfd17f2a0-kube-api-access-ghxpv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:45.709132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.709006    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:45.709132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.709016    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b391c51-7998-423f-bc7d-412cfd17f2a0-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:45.709132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.709025    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b391c51-7998-423f-bc7d-412cfd17f2a0-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:45.714582 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.714552    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"]
Nov 01 14:49:45.714851 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.714819    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm" podUID="ceda76db-81d2-4b69-abe2-5121ffe7d88b" containerName="payload-pre-processing" containerID="cri-o://2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6" gracePeriod=30
Nov 01 14:49:45.959280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:45.959249    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:49:46.001853 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.001825    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:49:46.012877 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.012848    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-plugins-config-volume\") pod \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\" (UID: \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\") "
Nov 01 14:49:46.013051 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.013027    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-kube-api-access-4mcvk\" (UniqueName: \"kubernetes.io/projected/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-kube-api-access-4mcvk\") pod \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\" (UID: \"65a92e04-f8d0-4bc8-9fcd-33dd819603f0\") "
Nov 01 14:49:46.013264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.013243    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-plugins-config-volume" pod "65a92e04-f8d0-4bc8-9fcd-33dd819603f0" (UID: "65a92e04-f8d0-4bc8-9fcd-33dd819603f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:49:46.013504 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.013486    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:46.015133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.015113    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-kube-api-access-4mcvk" pod "65a92e04-f8d0-4bc8-9fcd-33dd819603f0" (UID: "65a92e04-f8d0-4bc8-9fcd-33dd819603f0"). InnerVolumeSpecName "kube-api-access-4mcvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:49:46.114438 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.114401    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceda76db-81d2-4b69-abe2-5121ffe7d88b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceda76db-81d2-4b69-abe2-5121ffe7d88b-plugins-config-volume\") pod \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\" (UID: \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\") "
Nov 01 14:49:46.114642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.114458    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceda76db-81d2-4b69-abe2-5121ffe7d88b-kube-api-access-zsf67\" (UniqueName: \"kubernetes.io/projected/ceda76db-81d2-4b69-abe2-5121ffe7d88b-kube-api-access-zsf67\") pod \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\" (UID: \"ceda76db-81d2-4b69-abe2-5121ffe7d88b\") "
Nov 01 14:49:46.114807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.114781    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceda76db-81d2-4b69-abe2-5121ffe7d88b-plugins-config-volume" pod "ceda76db-81d2-4b69-abe2-5121ffe7d88b" (UID: "ceda76db-81d2-4b69-abe2-5121ffe7d88b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:49:46.115057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.115036    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mcvk\" (UniqueName: \"kubernetes.io/projected/65a92e04-f8d0-4bc8-9fcd-33dd819603f0-kube-api-access-4mcvk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:46.115156 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.115059    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceda76db-81d2-4b69-abe2-5121ffe7d88b-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:46.117083 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.117058    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceda76db-81d2-4b69-abe2-5121ffe7d88b-kube-api-access-zsf67" pod "ceda76db-81d2-4b69-abe2-5121ffe7d88b" (UID: "ceda76db-81d2-4b69-abe2-5121ffe7d88b"). InnerVolumeSpecName "kube-api-access-zsf67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:49:46.217885 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.217838    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsf67\" (UniqueName: \"kubernetes.io/projected/ceda76db-81d2-4b69-abe2-5121ffe7d88b-kube-api-access-zsf67\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:46.408518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.408486    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"
Nov 01 14:49:46.408700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.408522    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x" event={"ID":"9b391c51-7998-423f-bc7d-412cfd17f2a0","Type":"ContainerDied","Data":"9f9803580fad1e543ecb0b90e30445c678f6dbc102b0453403952a4a13ab9fd8"}
Nov 01 14:49:46.408700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.408585    5423 scope.go:122] "RemoveContainer" containerID="dd421d26859f6c6ca58f16b2d5a0b3b513d149fbe93e8665c52b5b44b29198a8"
Nov 01 14:49:46.410231 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.410205    5423 generic.go:356] "Generic (PLEG): container finished" podID="ceda76db-81d2-4b69-abe2-5121ffe7d88b" containerID="2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6" exitCode=0
Nov 01 14:49:46.410361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.410251    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"
Nov 01 14:49:46.410361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.410294    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm" event={"ID":"ceda76db-81d2-4b69-abe2-5121ffe7d88b","Type":"ContainerDied","Data":"2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6"}
Nov 01 14:49:46.410361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.410347    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm" event={"ID":"ceda76db-81d2-4b69-abe2-5121ffe7d88b","Type":"ContainerDied","Data":"9e8cc8235e8c8e2380a2df7b8a6c0bca1d4ad06c5f39c406658d6df1d60f805d"}
Nov 01 14:49:46.412930 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.412905    5423 generic.go:356] "Generic (PLEG): container finished" podID="65a92e04-f8d0-4bc8-9fcd-33dd819603f0" containerID="de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81" exitCode=0
Nov 01 14:49:46.413030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.412935    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s" event={"ID":"65a92e04-f8d0-4bc8-9fcd-33dd819603f0","Type":"ContainerDied","Data":"de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81"}
Nov 01 14:49:46.413030 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.412970    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s" event={"ID":"65a92e04-f8d0-4bc8-9fcd-33dd819603f0","Type":"ContainerDied","Data":"31820f3b681b2869d929d8d39cf454c1f3606ef93ccd32ba82d4c30889279504"}
Nov 01 14:49:46.413216 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.413192    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"
Nov 01 14:49:46.418689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.418672    5423 scope.go:122] "RemoveContainer" containerID="2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6"
Nov 01 14:49:46.427755 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.427733    5423 scope.go:122] "RemoveContainer" containerID="2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6"
Nov 01 14:49:46.428030 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:49:46.428012    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6\": container with ID starting with 2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6 not found: ID does not exist" containerID="2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6"
Nov 01 14:49:46.428104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.428038    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6"} err="failed to get container status \"2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6\": rpc error: code = NotFound desc = could not find container \"2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6\": container with ID starting with 2ee1f89e354e4ef5fb5baec2a1e636c560ece872f5ca4678b446387efdd15bf6 not found: ID does not exist"
Nov 01 14:49:46.428104 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.428055    5423 scope.go:122] "RemoveContainer" containerID="de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81"
Nov 01 14:49:46.436130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.436114    5423 scope.go:122] "RemoveContainer" containerID="de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81"
Nov 01 14:49:46.436496 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:49:46.436367    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81\": container with ID starting with de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81 not found: ID does not exist" containerID="de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81"
Nov 01 14:49:46.436496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.436411    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81"} err="failed to get container status \"de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81\": rpc error: code = NotFound desc = could not find container \"de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81\": container with ID starting with de21d7271cc4d374de2cd3e7cfa07e032eb3b5634217d47709a0315bced69b81 not found: ID does not exist"
Nov 01 14:49:46.576400 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.576364    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"]
Nov 01 14:49:46.582179 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.582154    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86d8df8f44-nr68s"]
Nov 01 14:49:46.694429 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.694350    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"]
Nov 01 14:49:46.698101 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.698079    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25ce0b-7897977d7f-9mq5x"]
Nov 01 14:49:46.808786 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.808759    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"]
Nov 01 14:49:46.811720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.811696    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm"]
Nov 01 14:49:46.976753 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:46.976675    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6985c85d79-p4kcm" podUID="ceda76db-81d2-4b69-abe2-5121ffe7d88b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.114:9004: i/o timeout"
Nov 01 14:49:47.346092 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:47.346065    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:49:47.962380 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:47.962346    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a92e04-f8d0-4bc8-9fcd-33dd819603f0" path="/var/lib/kubelet/pods/65a92e04-f8d0-4bc8-9fcd-33dd819603f0/volumes"
Nov 01 14:49:47.962888 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:47.962865    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b391c51-7998-423f-bc7d-412cfd17f2a0" path="/var/lib/kubelet/pods/9b391c51-7998-423f-bc7d-412cfd17f2a0/volumes"
Nov 01 14:49:47.963316 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:47.963295    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceda76db-81d2-4b69-abe2-5121ffe7d88b" path="/var/lib/kubelet/pods/ceda76db-81d2-4b69-abe2-5121ffe7d88b/volumes"
Nov 01 14:49:48.354077 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:48.354049    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:49:52.189979 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.189946    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"]
Nov 01 14:49:52.190553 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190536    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9b391c51-7998-423f-bc7d-412cfd17f2a0" containerName="maas-api"
Nov 01 14:49:52.190603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190554    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b391c51-7998-423f-bc7d-412cfd17f2a0" containerName="maas-api"
Nov 01 14:49:52.190603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190585    5423 cpu_manager.go:414] "containerMap: removing container" podUID="65a92e04-f8d0-4bc8-9fcd-33dd819603f0" containerName="payload-processing"
Nov 01 14:49:52.190603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190591    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a92e04-f8d0-4bc8-9fcd-33dd819603f0" containerName="payload-processing"
Nov 01 14:49:52.190603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190602    5423 cpu_manager.go:414] "containerMap: removing container" podUID="ceda76db-81d2-4b69-abe2-5121ffe7d88b" containerName="payload-pre-processing"
Nov 01 14:49:52.190728 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190608    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceda76db-81d2-4b69-abe2-5121ffe7d88b" containerName="payload-pre-processing"
Nov 01 14:49:52.190728 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190688    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a92e04-f8d0-4bc8-9fcd-33dd819603f0" containerName="payload-processing"
Nov 01 14:49:52.190728 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190697    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceda76db-81d2-4b69-abe2-5121ffe7d88b" containerName="payload-pre-processing"
Nov 01 14:49:52.190728 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.190709    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b391c51-7998-423f-bc7d-412cfd17f2a0" containerName="maas-api"
Nov 01 14:49:52.195464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.195445    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:52.247625 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.247597    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"]
Nov 01 14:49:52.294987 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.294958    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7c335a-dockercfg-hpqps\""
Nov 01 14:49:52.295132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.295044    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7c335a\""
Nov 01 14:49:52.310838 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.310803    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079a6430-8709-4bf3-a0b2-ff8489cf56a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9\" (UID: \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:52.310969 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.310912    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chv9b\" (UniqueName: \"kubernetes.io/projected/079a6430-8709-4bf3-a0b2-ff8489cf56a3-kube-api-access-chv9b\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9\" (UID: \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:52.412695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.412659    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv9b\" (UniqueName: \"kubernetes.io/projected/079a6430-8709-4bf3-a0b2-ff8489cf56a3-kube-api-access-chv9b\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9\" (UID: \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:52.412865 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.412818    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079a6430-8709-4bf3-a0b2-ff8489cf56a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9\" (UID: \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:52.413561 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.413540    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079a6430-8709-4bf3-a0b2-ff8489cf56a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9\" (UID: \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:52.509191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.509113    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv9b\" (UniqueName: \"kubernetes.io/projected/079a6430-8709-4bf3-a0b2-ff8489cf56a3-kube-api-access-chv9b\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9\" (UID: \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:52.509522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.509493    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:52.887318 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:49:52.887284    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079a6430_8709_4bf3_a0b2_ff8489cf56a3.slice/crio-2e7ca2b566bcff7fd180d60e2a11ac4b05c0f81896fe14008b5a0b66cbe0d70c WatchSource:0}: Error finding container 2e7ca2b566bcff7fd180d60e2a11ac4b05c0f81896fe14008b5a0b66cbe0d70c: Status 404 returned error can't find the container with id 2e7ca2b566bcff7fd180d60e2a11ac4b05c0f81896fe14008b5a0b66cbe0d70c
Nov 01 14:49:52.890527 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.890481    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"]
Nov 01 14:49:52.966723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.966700    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"]
Nov 01 14:49:52.970881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:52.970862    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:53.023890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.023857    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"]
Nov 01 14:49:53.120515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.120484    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgf6s\" (UniqueName: \"kubernetes.io/projected/9e834dc1-573f-4c1a-b252-898425b81872-kube-api-access-rgf6s\") pod \"payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99\" (UID: \"9e834dc1-573f-4c1a-b252-898425b81872\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:53.120691 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.120595    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e834dc1-573f-4c1a-b252-898425b81872-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99\" (UID: \"9e834dc1-573f-4c1a-b252-898425b81872\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:53.221766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.221693    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgf6s\" (UniqueName: \"kubernetes.io/projected/9e834dc1-573f-4c1a-b252-898425b81872-kube-api-access-rgf6s\") pod \"payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99\" (UID: \"9e834dc1-573f-4c1a-b252-898425b81872\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:53.222236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.221777    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e834dc1-573f-4c1a-b252-898425b81872-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99\" (UID: \"9e834dc1-573f-4c1a-b252-898425b81872\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:53.222616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.222591    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e834dc1-573f-4c1a-b252-898425b81872-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99\" (UID: \"9e834dc1-573f-4c1a-b252-898425b81872\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:53.280000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.279972    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgf6s\" (UniqueName: \"kubernetes.io/projected/9e834dc1-573f-4c1a-b252-898425b81872-kube-api-access-rgf6s\") pod \"payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99\" (UID: \"9e834dc1-573f-4c1a-b252-898425b81872\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:53.288352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.288329    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:53.488554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.488478    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9" event={"ID":"079a6430-8709-4bf3-a0b2-ff8489cf56a3","Type":"ContainerStarted","Data":"ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23"}
Nov 01 14:49:53.488554 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.488532    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9" event={"ID":"079a6430-8709-4bf3-a0b2-ff8489cf56a3","Type":"ContainerStarted","Data":"2e7ca2b566bcff7fd180d60e2a11ac4b05c0f81896fe14008b5a0b66cbe0d70c"}
Nov 01 14:49:53.488768 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.488589    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:49:53.606653 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.606594    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9" podStartSLOduration=1.60658046 podStartE2EDuration="1.60658046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:53.604836334 +0000 UTC m=+1933.770321621" watchObservedRunningTime="2026-11-01 14:49:53.60658046 +0000 UTC m=+1933.772065759"
Nov 01 14:49:53.709972 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:49:53.708984    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e834dc1_573f_4c1a_b252_898425b81872.slice/crio-79d3ff04983afacc246d7b822951abb698984d6e10bca7a1930382527123b06b WatchSource:0}: Error finding container 79d3ff04983afacc246d7b822951abb698984d6e10bca7a1930382527123b06b: Status 404 returned error can't find the container with id 79d3ff04983afacc246d7b822951abb698984d6e10bca7a1930382527123b06b
Nov 01 14:49:53.712912 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:53.712883    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"]
Nov 01 14:49:54.495981 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:54.495943    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99" event={"ID":"9e834dc1-573f-4c1a-b252-898425b81872","Type":"ContainerStarted","Data":"3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130"}
Nov 01 14:49:54.496466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:54.496027    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99" event={"ID":"9e834dc1-573f-4c1a-b252-898425b81872","Type":"ContainerStarted","Data":"79d3ff04983afacc246d7b822951abb698984d6e10bca7a1930382527123b06b"}
Nov 01 14:49:54.614158 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:54.614110    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99" podStartSLOduration=2.614094911 podStartE2EDuration="2.614094911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:54.611858801 +0000 UTC m=+1934.777344101" watchObservedRunningTime="2026-11-01 14:49:54.614094911 +0000 UTC m=+1934.779580212"
Nov 01 14:49:55.501636 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:55.501595    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:49:56.091926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:49:56.091886    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:50:04.499239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.499205    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:50:04.886835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.886804    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"]
Nov 01 14:50:04.890842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.890822    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:04.907899 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.907875    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zjp\" (UniqueName: \"kubernetes.io/projected/493c4113-cb63-48dc-b340-c2ca52d80cae-kube-api-access-f6zjp\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:04.908014 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.907995    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493c4113-cb63-48dc-b340-c2ca52d80cae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:04.908103 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.908084    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-metrics-certs\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:04.908191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.908177    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:04.943996 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.943973    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d195c1-serving-cert\""
Nov 01 14:50:04.944518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:04.944482    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"]
Nov 01 14:50:05.009727 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.009697    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493c4113-cb63-48dc-b340-c2ca52d80cae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.009869 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.009760    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-metrics-certs\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.010106 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.010081    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.010995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.010910    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zjp\" (UniqueName: \"kubernetes.io/projected/493c4113-cb63-48dc-b340-c2ca52d80cae-kube-api-access-f6zjp\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.011286 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:05.010319    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-serving-cert: secret "maas-api-e2e-praxis-a-d195c1-serving-cert" not found
Nov 01 14:50:05.011378 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:05.011342    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls podName:493c4113-cb63-48dc-b340-c2ca52d80cae nodeName:}" failed. No retries permitted until 2026-11-01 14:50:05.511321657 +0000 UTC m=+1945.676806945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls") pod "maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc" (UID: "493c4113-cb63-48dc-b340-c2ca52d80cae") : secret "maas-api-e2e-praxis-a-d195c1-serving-cert" not found
Nov 01 14:50:05.012026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.010770    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493c4113-cb63-48dc-b340-c2ca52d80cae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.012775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.012750    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-metrics-certs\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.072412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.072368    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zjp\" (UniqueName: \"kubernetes.io/projected/493c4113-cb63-48dc-b340-c2ca52d80cae-kube-api-access-f6zjp\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.204106 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:05.204020    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:50:05.518373 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.518294    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.520698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.520675    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:05.804266 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:05.804191    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:06.173474 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:50:06.173442    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493c4113_cb63_48dc_b340_c2ca52d80cae.slice/crio-957ba20e416d242c910281bae8e5eae5b58a549d363900d3eee3663bf1ca3232 WatchSource:0}: Error finding container 957ba20e416d242c910281bae8e5eae5b58a549d363900d3eee3663bf1ca3232: Status 404 returned error can't find the container with id 957ba20e416d242c910281bae8e5eae5b58a549d363900d3eee3663bf1ca3232
Nov 01 14:50:06.175611 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:06.175591    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:50:06.175889 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:06.175865    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"]
Nov 01 14:50:06.509291 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:06.509215    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:50:06.634484 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:06.634442    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc" event={"ID":"493c4113-cb63-48dc-b340-c2ca52d80cae","Type":"ContainerStarted","Data":"957ba20e416d242c910281bae8e5eae5b58a549d363900d3eee3663bf1ca3232"}
Nov 01 14:50:07.717195 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:07.717148    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc" event={"ID":"493c4113-cb63-48dc-b340-c2ca52d80cae","Type":"ContainerStarted","Data":"f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46"}
Nov 01 14:50:07.717783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:07.717552    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:07.837256 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:07.837193    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc" podStartSLOduration=2.87608335 podStartE2EDuration="3.837173815s" totalImagesPullingTime="961.090465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:50:07.835754732 +0000 UTC m=+1948.001240032" watchObservedRunningTime="2026-11-01 14:50:07.837173815 +0000 UTC m=+1948.002659114"
Nov 01 14:50:13.728364 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:13.728328    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:50:23.414412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.405034    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"]
Nov 01 14:50:23.414412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.405292    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l" podUID="78378ebb-07db-47e6-8f2a-bc89d6c3d838" containerName="payload-processing" containerID="cri-o://0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c" gracePeriod=30
Nov 01 14:50:23.459474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.459444    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"]
Nov 01 14:50:23.459677 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.459655    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf" podUID="3abbebb1-3e50-414c-b641-c9d66e824ce7" containerName="payload-pre-processing" containerID="cri-o://6646508937ea9372a19c7aba9f351cd871d936b2041c268449686e51489984ff" gracePeriod=30
Nov 01 14:50:23.486158 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:23.485194    5423 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78378ebb_07db_47e6_8f2a_bc89d6c3d838.slice/crio-0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 14:50:23.771964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.771938    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:50:23.844412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.844365    5423 generic.go:356] "Generic (PLEG): container finished" podID="3abbebb1-3e50-414c-b641-c9d66e824ce7" containerID="6646508937ea9372a19c7aba9f351cd871d936b2041c268449686e51489984ff" exitCode=0
Nov 01 14:50:23.844552 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.844439    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf" event={"ID":"3abbebb1-3e50-414c-b641-c9d66e824ce7","Type":"ContainerDied","Data":"6646508937ea9372a19c7aba9f351cd871d936b2041c268449686e51489984ff"}
Nov 01 14:50:23.844552 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.844496    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf" event={"ID":"3abbebb1-3e50-414c-b641-c9d66e824ce7","Type":"ContainerDied","Data":"c3f212279c8c727eede68e9573a8f8b0b46244994f3b018921cbda6d90dccf67"}
Nov 01 14:50:23.844552 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.844522    5423 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3f212279c8c727eede68e9573a8f8b0b46244994f3b018921cbda6d90dccf67"
Nov 01 14:50:23.845900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.845876    5423 generic.go:356] "Generic (PLEG): container finished" podID="78378ebb-07db-47e6-8f2a-bc89d6c3d838" containerID="0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c" exitCode=0
Nov 01 14:50:23.846037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.845936    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"
Nov 01 14:50:23.846037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.845948    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l" event={"ID":"78378ebb-07db-47e6-8f2a-bc89d6c3d838","Type":"ContainerDied","Data":"0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c"}
Nov 01 14:50:23.846037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.845981    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l" event={"ID":"78378ebb-07db-47e6-8f2a-bc89d6c3d838","Type":"ContainerDied","Data":"13e7801c4f915cf194055b22ef18990140f15a4563c426237cf06efbafba0d15"}
Nov 01 14:50:23.846037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.846000    5423 scope.go:122] "RemoveContainer" containerID="0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c"
Nov 01 14:50:23.850766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.850738    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78378ebb-07db-47e6-8f2a-bc89d6c3d838-kube-api-access-cmk7t\" (UniqueName: \"kubernetes.io/projected/78378ebb-07db-47e6-8f2a-bc89d6c3d838-kube-api-access-cmk7t\") pod \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\" (UID: \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\") "
Nov 01 14:50:23.850919 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.850898    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78378ebb-07db-47e6-8f2a-bc89d6c3d838-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78378ebb-07db-47e6-8f2a-bc89d6c3d838-plugins-config-volume\") pod \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\" (UID: \"78378ebb-07db-47e6-8f2a-bc89d6c3d838\") "
Nov 01 14:50:23.851251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.851228    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78378ebb-07db-47e6-8f2a-bc89d6c3d838-plugins-config-volume" pod "78378ebb-07db-47e6-8f2a-bc89d6c3d838" (UID: "78378ebb-07db-47e6-8f2a-bc89d6c3d838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:23.851482 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.851437    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78378ebb-07db-47e6-8f2a-bc89d6c3d838-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:23.852771 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.852756    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:50:23.853233 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.853216    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78378ebb-07db-47e6-8f2a-bc89d6c3d838-kube-api-access-cmk7t" pod "78378ebb-07db-47e6-8f2a-bc89d6c3d838" (UID: "78378ebb-07db-47e6-8f2a-bc89d6c3d838"). InnerVolumeSpecName "kube-api-access-cmk7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:23.856333 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.856315    5423 scope.go:122] "RemoveContainer" containerID="0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c"
Nov 01 14:50:23.856603 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:23.856579    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c\": container with ID starting with 0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c not found: ID does not exist" containerID="0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c"
Nov 01 14:50:23.856689 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:23.856609    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c"} err="failed to get container status \"0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c\": rpc error: code = NotFound desc = could not find container \"0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c\": container with ID starting with 0a909807550decea5a6a0dfec8b280e0c5849599fa91630f11f219286415254c not found: ID does not exist"
Nov 01 14:50:24.058284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.058246    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3abbebb1-3e50-414c-b641-c9d66e824ce7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abbebb1-3e50-414c-b641-c9d66e824ce7-plugins-config-volume\") pod \"3abbebb1-3e50-414c-b641-c9d66e824ce7\" (UID: \"3abbebb1-3e50-414c-b641-c9d66e824ce7\") "
Nov 01 14:50:24.058908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.058881    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abbebb1-3e50-414c-b641-c9d66e824ce7-plugins-config-volume" pod "3abbebb1-3e50-414c-b641-c9d66e824ce7" (UID: "3abbebb1-3e50-414c-b641-c9d66e824ce7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:24.058973 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.058443    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abbebb1-3e50-414c-b641-c9d66e824ce7-kube-api-access-sqhbd\" (UniqueName: \"kubernetes.io/projected/3abbebb1-3e50-414c-b641-c9d66e824ce7-kube-api-access-sqhbd\") pod \"3abbebb1-3e50-414c-b641-c9d66e824ce7\" (UID: \"3abbebb1-3e50-414c-b641-c9d66e824ce7\") "
Nov 01 14:50:24.059500 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.059476    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmk7t\" (UniqueName: \"kubernetes.io/projected/78378ebb-07db-47e6-8f2a-bc89d6c3d838-kube-api-access-cmk7t\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:24.059561 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.059511    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abbebb1-3e50-414c-b641-c9d66e824ce7-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:24.069795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.069743    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abbebb1-3e50-414c-b641-c9d66e824ce7-kube-api-access-sqhbd" pod "3abbebb1-3e50-414c-b641-c9d66e824ce7" (UID: "3abbebb1-3e50-414c-b641-c9d66e824ce7"). InnerVolumeSpecName "kube-api-access-sqhbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:24.161062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.161024    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqhbd\" (UniqueName: \"kubernetes.io/projected/3abbebb1-3e50-414c-b641-c9d66e824ce7-kube-api-access-sqhbd\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:24.280944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.280905    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"]
Nov 01 14:50:24.283670 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.283649    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l"]
Nov 01 14:50:24.661628 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.661591    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-757675f65d-zt95l" podUID="78378ebb-07db-47e6-8f2a-bc89d6c3d838" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.113:9004: i/o timeout"
Nov 01 14:50:24.852375 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:24.852341    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"
Nov 01 14:50:25.038531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:25.038252    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"]
Nov 01 14:50:25.049115 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:25.049082    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-9f855568c-6ldpf"]
Nov 01 14:50:25.961515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:25.961484    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abbebb1-3e50-414c-b641-c9d66e824ce7" path="/var/lib/kubelet/pods/3abbebb1-3e50-414c-b641-c9d66e824ce7/volumes"
Nov 01 14:50:25.961891 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:25.961883    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78378ebb-07db-47e6-8f2a-bc89d6c3d838" path="/var/lib/kubelet/pods/78378ebb-07db-47e6-8f2a-bc89d6c3d838/volumes"
Nov 01 14:50:29.890031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:29.889999    5423 generic.go:356] "Generic (PLEG): container finished" podID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerID="a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f" exitCode=0
Nov 01 14:50:29.890472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:29.890070    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" event={"ID":"8ec6d420-5b16-4a0b-868c-819eea8a7368","Type":"ContainerDied","Data":"a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f"}
Nov 01 14:50:29.890520 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:29.890481    5423 scope.go:122] "RemoveContainer" containerID="a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f"
Nov 01 14:50:29.892270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:29.892235    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:50:29.892380 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:29.892328    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:50:29.892380 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:29.892377    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:50:30.024712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.024678    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"]
Nov 01 14:50:30.025508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.025488    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3abbebb1-3e50-414c-b641-c9d66e824ce7" containerName="payload-pre-processing"
Nov 01 14:50:30.025508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.025507    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abbebb1-3e50-414c-b641-c9d66e824ce7" containerName="payload-pre-processing"
Nov 01 14:50:30.025693 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.025541    5423 cpu_manager.go:414] "containerMap: removing container" podUID="78378ebb-07db-47e6-8f2a-bc89d6c3d838" containerName="payload-processing"
Nov 01 14:50:30.025693 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.025550    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78378ebb-07db-47e6-8f2a-bc89d6c3d838" containerName="payload-processing"
Nov 01 14:50:30.025693 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.025675    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="78378ebb-07db-47e6-8f2a-bc89d6c3d838" containerName="payload-processing"
Nov 01 14:50:30.025860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.025696    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abbebb1-3e50-414c-b641-c9d66e824ce7" containerName="payload-pre-processing"
Nov 01 14:50:30.028424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.028375    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:30.109535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.109499    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"]
Nov 01 14:50:30.123422 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.114599    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2048c1\""
Nov 01 14:50:30.123422 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.114646    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2048c1-dockercfg-w88k5\""
Nov 01 14:50:30.144264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.144150    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dspx4\" (UniqueName: \"kubernetes.io/projected/d31e1668-0499-44ad-9c64-4852f7d1a3c8-kube-api-access-dspx4\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w\" (UID: \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:30.144443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.144378    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1668-0499-44ad-9c64-4852f7d1a3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w\" (UID: \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:30.284456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.247570    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dspx4\" (UniqueName: \"kubernetes.io/projected/d31e1668-0499-44ad-9c64-4852f7d1a3c8-kube-api-access-dspx4\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w\" (UID: \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:30.284456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.247723    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1668-0499-44ad-9c64-4852f7d1a3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w\" (UID: \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:30.284456 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.248737    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1668-0499-44ad-9c64-4852f7d1a3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w\" (UID: \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:30.352784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.352749    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dspx4\" (UniqueName: \"kubernetes.io/projected/d31e1668-0499-44ad-9c64-4852f7d1a3c8-kube-api-access-dspx4\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w\" (UID: \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:30.643189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.643150    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:30.926507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.926419    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" event={"ID":"8ec6d420-5b16-4a0b-868c-819eea8a7368","Type":"ContainerStarted","Data":"37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103"}
Nov 01 14:50:30.975274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.975245    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:50:30.975449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.975363    5423 patch_prober.go:29] interesting pod/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 14:50:30.975449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:30.975417    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 14:50:31.039695 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:50:31.039660    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31e1668_0499_44ad_9c64_4852f7d1a3c8.slice/crio-4cd85aae3bfa38467f4a2e011f263d3b94f2af5c8516c36e0484313a02df66c1 WatchSource:0}: Error finding container 4cd85aae3bfa38467f4a2e011f263d3b94f2af5c8516c36e0484313a02df66c1: Status 404 returned error can't find the container with id 4cd85aae3bfa38467f4a2e011f263d3b94f2af5c8516c36e0484313a02df66c1
Nov 01 14:50:31.042855 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:31.042831    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"]
Nov 01 14:50:31.933614 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:31.933573    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w" event={"ID":"d31e1668-0499-44ad-9c64-4852f7d1a3c8","Type":"ContainerStarted","Data":"54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d"}
Nov 01 14:50:31.934031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:31.933720    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:31.934031 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:31.933757    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w" event={"ID":"d31e1668-0499-44ad-9c64-4852f7d1a3c8","Type":"ContainerStarted","Data":"4cd85aae3bfa38467f4a2e011f263d3b94f2af5c8516c36e0484313a02df66c1"}
Nov 01 14:50:31.975188 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:31.975156    5423 patch_prober.go:29] interesting pod/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 14:50:31.975342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:31.975204    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 14:50:32.050358 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:32.050304    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w" podStartSLOduration=3.050290315 podStartE2EDuration="3.050290315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:50:32.049626339 +0000 UTC m=+1972.215111648" watchObservedRunningTime="2026-11-01 14:50:32.050290315 +0000 UTC m=+1972.215775612"
Nov 01 14:50:32.975057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:32.975023    5423 patch_prober.go:29] interesting pod/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 14:50:32.975511 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:32.975078    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 14:50:33.975235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:33.975199    5423 patch_prober.go:29] interesting pod/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 14:50:33.975669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:33.975250    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 14:50:34.975325 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:34.975288    5423 patch_prober.go:29] interesting pod/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 14:50:34.975724 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:34.975341    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 14:50:35.150493 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:35.150455    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:50:35.975340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:35.975304    5423 patch_prober.go:29] interesting pod/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 14:50:35.975755 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:35.975354    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 14:50:36.077437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:36.077400    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:50:36.979518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:36.979492    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:50:36.980525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:36.980506    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:50:40.514357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.514322    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"]
Nov 01 14:50:40.514805 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.514521    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c" podUID="f4e8547b-a373-4e35-95fa-fe3844ae2ff6" containerName="maas-api" containerID="cri-o://a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234" gracePeriod=30
Nov 01 14:50:40.848277 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.848248    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:50:40.941439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.940652    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-openshift-service-ca\") pod \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") "
Nov 01 14:50:40.941439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.940783    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-metrics-certs\") pod \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") "
Nov 01 14:50:40.941439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.940817    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-kube-api-access-2ss7c\" (UniqueName: \"kubernetes.io/projected/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-kube-api-access-2ss7c\") pod \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") "
Nov 01 14:50:40.941439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.940918    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls\") pod \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\" (UID: \"f4e8547b-a373-4e35-95fa-fe3844ae2ff6\") "
Nov 01 14:50:40.942934 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.942877    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-openshift-service-ca" pod "f4e8547b-a373-4e35-95fa-fe3844ae2ff6" (UID: "f4e8547b-a373-4e35-95fa-fe3844ae2ff6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:40.949214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.949184    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-metrics-certs" pod "f4e8547b-a373-4e35-95fa-fe3844ae2ff6" (UID: "f4e8547b-a373-4e35-95fa-fe3844ae2ff6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:50:40.952352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.952308    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls" pod "f4e8547b-a373-4e35-95fa-fe3844ae2ff6" (UID: "f4e8547b-a373-4e35-95fa-fe3844ae2ff6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:50:40.963535 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.960444    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-kube-api-access-2ss7c" pod "f4e8547b-a373-4e35-95fa-fe3844ae2ff6" (UID: "f4e8547b-a373-4e35-95fa-fe3844ae2ff6"). InnerVolumeSpecName "kube-api-access-2ss7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:40.996319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.996287    5423 generic.go:356] "Generic (PLEG): container finished" podID="f4e8547b-a373-4e35-95fa-fe3844ae2ff6" containerID="a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234" exitCode=0
Nov 01 14:50:40.996579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.996368    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c" event={"ID":"f4e8547b-a373-4e35-95fa-fe3844ae2ff6","Type":"ContainerDied","Data":"a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234"}
Nov 01 14:50:40.996810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.996782    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c" event={"ID":"f4e8547b-a373-4e35-95fa-fe3844ae2ff6","Type":"ContainerDied","Data":"de57080eb3261d1b8ed67e5dc668ea60de84ba4ade2b85e54fbb18dc9fb87783"}
Nov 01 14:50:40.996927 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.996836    5423 scope.go:122] "RemoveContainer" containerID="a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234"
Nov 01 14:50:40.996987 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:40.996439    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"
Nov 01 14:50:41.009868 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.009852    5423 scope.go:122] "RemoveContainer" containerID="a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234"
Nov 01 14:50:41.010123 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:41.010103    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234\": container with ID starting with a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234 not found: ID does not exist" containerID="a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234"
Nov 01 14:50:41.010187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.010129    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234"} err="failed to get container status \"a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234\": rpc error: code = NotFound desc = could not find container \"a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234\": container with ID starting with a7d11eb5e2e982052232ffcc453182ad09ca6037eaabad70fd4a19a59dd04234 not found: ID does not exist"
Nov 01 14:50:41.023896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.023871    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"]
Nov 01 14:50:41.024048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.024031    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99" podUID="9e834dc1-573f-4c1a-b252-898425b81872" containerName="payload-processing" containerID="cri-o://3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130" gracePeriod=30
Nov 01 14:50:41.042841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.042817    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:41.042942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.042848    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:41.042942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.042859    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ss7c\" (UniqueName: \"kubernetes.io/projected/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-kube-api-access-2ss7c\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:41.042942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.042869    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4e8547b-a373-4e35-95fa-fe3844ae2ff6-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:41.073357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.073333    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"]
Nov 01 14:50:41.073574 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.073550    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9" podUID="079a6430-8709-4bf3-a0b2-ff8489cf56a3" containerName="payload-pre-processing" containerID="cri-o://ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23" gracePeriod=30
Nov 01 14:50:41.294242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.294204    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"]
Nov 01 14:50:41.296613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.296586    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c335a-849f46b4d9-9bl4c"]
Nov 01 14:50:41.325413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.320865    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:50:41.376441 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.376414    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:50:41.455627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.455600    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079a6430-8709-4bf3-a0b2-ff8489cf56a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079a6430-8709-4bf3-a0b2-ff8489cf56a3-plugins-config-volume\") pod \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\" (UID: \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\") "
Nov 01 14:50:41.455753 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.455639    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079a6430-8709-4bf3-a0b2-ff8489cf56a3-kube-api-access-chv9b\" (UniqueName: \"kubernetes.io/projected/079a6430-8709-4bf3-a0b2-ff8489cf56a3-kube-api-access-chv9b\") pod \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\" (UID: \"079a6430-8709-4bf3-a0b2-ff8489cf56a3\") "
Nov 01 14:50:41.455808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.455760    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e834dc1-573f-4c1a-b252-898425b81872-kube-api-access-rgf6s\" (UniqueName: \"kubernetes.io/projected/9e834dc1-573f-4c1a-b252-898425b81872-kube-api-access-rgf6s\") pod \"9e834dc1-573f-4c1a-b252-898425b81872\" (UID: \"9e834dc1-573f-4c1a-b252-898425b81872\") "
Nov 01 14:50:41.455846 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.455823    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e834dc1-573f-4c1a-b252-898425b81872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e834dc1-573f-4c1a-b252-898425b81872-plugins-config-volume\") pod \"9e834dc1-573f-4c1a-b252-898425b81872\" (UID: \"9e834dc1-573f-4c1a-b252-898425b81872\") "
Nov 01 14:50:41.456091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.456031    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079a6430-8709-4bf3-a0b2-ff8489cf56a3-plugins-config-volume" pod "079a6430-8709-4bf3-a0b2-ff8489cf56a3" (UID: "079a6430-8709-4bf3-a0b2-ff8489cf56a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:41.456235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.456216    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e834dc1-573f-4c1a-b252-898425b81872-plugins-config-volume" pod "9e834dc1-573f-4c1a-b252-898425b81872" (UID: "9e834dc1-573f-4c1a-b252-898425b81872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:41.456293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.456224    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/079a6430-8709-4bf3-a0b2-ff8489cf56a3-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:41.457875 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.457855    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e834dc1-573f-4c1a-b252-898425b81872-kube-api-access-rgf6s" pod "9e834dc1-573f-4c1a-b252-898425b81872" (UID: "9e834dc1-573f-4c1a-b252-898425b81872"). InnerVolumeSpecName "kube-api-access-rgf6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:41.457937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.457887    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079a6430-8709-4bf3-a0b2-ff8489cf56a3-kube-api-access-chv9b" pod "079a6430-8709-4bf3-a0b2-ff8489cf56a3" (UID: "079a6430-8709-4bf3-a0b2-ff8489cf56a3"). InnerVolumeSpecName "kube-api-access-chv9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:41.566468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.563441    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgf6s\" (UniqueName: \"kubernetes.io/projected/9e834dc1-573f-4c1a-b252-898425b81872-kube-api-access-rgf6s\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:41.566468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.563483    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e834dc1-573f-4c1a-b252-898425b81872-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:41.566468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.563498    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chv9b\" (UniqueName: \"kubernetes.io/projected/079a6430-8709-4bf3-a0b2-ff8489cf56a3-kube-api-access-chv9b\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:41.960817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:41.960785    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e8547b-a373-4e35-95fa-fe3844ae2ff6" path="/var/lib/kubelet/pods/f4e8547b-a373-4e35-95fa-fe3844ae2ff6/volumes"
Nov 01 14:50:42.003276 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.003247    5423 generic.go:356] "Generic (PLEG): container finished" podID="9e834dc1-573f-4c1a-b252-898425b81872" containerID="3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130" exitCode=0
Nov 01 14:50:42.003418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.003301    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"
Nov 01 14:50:42.003418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.003325    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99" event={"ID":"9e834dc1-573f-4c1a-b252-898425b81872","Type":"ContainerDied","Data":"3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130"}
Nov 01 14:50:42.003418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.003371    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99" event={"ID":"9e834dc1-573f-4c1a-b252-898425b81872","Type":"ContainerDied","Data":"79d3ff04983afacc246d7b822951abb698984d6e10bca7a1930382527123b06b"}
Nov 01 14:50:42.003418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.003406    5423 scope.go:122] "RemoveContainer" containerID="3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130"
Nov 01 14:50:42.004868 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.004839    5423 generic.go:356] "Generic (PLEG): container finished" podID="079a6430-8709-4bf3-a0b2-ff8489cf56a3" containerID="ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23" exitCode=0
Nov 01 14:50:42.005010 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.004873    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9" event={"ID":"079a6430-8709-4bf3-a0b2-ff8489cf56a3","Type":"ContainerDied","Data":"ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23"}
Nov 01 14:50:42.005010 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.004901    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"
Nov 01 14:50:42.005133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.004926    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9" event={"ID":"079a6430-8709-4bf3-a0b2-ff8489cf56a3","Type":"ContainerDied","Data":"2e7ca2b566bcff7fd180d60e2a11ac4b05c0f81896fe14008b5a0b66cbe0d70c"}
Nov 01 14:50:42.012923 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.012906    5423 scope.go:122] "RemoveContainer" containerID="3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130"
Nov 01 14:50:42.013193 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:42.013173    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130\": container with ID starting with 3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130 not found: ID does not exist" containerID="3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130"
Nov 01 14:50:42.013243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.013205    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130"} err="failed to get container status \"3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130\": rpc error: code = NotFound desc = could not find container \"3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130\": container with ID starting with 3193fe25189413b900ff19aaaeff71a9c7b2a29ed6de6bea8e2550b3e43d8130 not found: ID does not exist"
Nov 01 14:50:42.013243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.013221    5423 scope.go:122] "RemoveContainer" containerID="ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23"
Nov 01 14:50:42.021530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.021502    5423 scope.go:122] "RemoveContainer" containerID="ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23"
Nov 01 14:50:42.021786 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:42.021766    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23\": container with ID starting with ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23 not found: ID does not exist" containerID="ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23"
Nov 01 14:50:42.021843 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.021794    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23"} err="failed to get container status \"ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23\": rpc error: code = NotFound desc = could not find container \"ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23\": container with ID starting with ad7406ae693861812a081af5238bff441195f588cd80ac3b4fed25af47d85d23 not found: ID does not exist"
Nov 01 14:50:42.169929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.169894    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"]
Nov 01 14:50:42.172085 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.172064    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c335a-54fbff7c4d-jjm99"]
Nov 01 14:50:42.283286 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.283195    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"]
Nov 01 14:50:42.285366 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.285345    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-7b74b97fc9-prhp9"]
Nov 01 14:50:42.942259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:42.942230    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:50:43.960500 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:43.960469    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079a6430-8709-4bf3-a0b2-ff8489cf56a3" path="/var/lib/kubelet/pods/079a6430-8709-4bf3-a0b2-ff8489cf56a3/volumes"
Nov 01 14:50:43.961008 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:43.960991    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e834dc1-573f-4c1a-b252-898425b81872" path="/var/lib/kubelet/pods/9e834dc1-573f-4c1a-b252-898425b81872/volumes"
Nov 01 14:50:44.835079 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:44.835048    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:50:47.244679 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.244646    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"]
Nov 01 14:50:47.245363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245341    5423 cpu_manager.go:414] "containerMap: removing container" podUID="079a6430-8709-4bf3-a0b2-ff8489cf56a3" containerName="payload-pre-processing"
Nov 01 14:50:47.245489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245364    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079a6430-8709-4bf3-a0b2-ff8489cf56a3" containerName="payload-pre-processing"
Nov 01 14:50:47.245489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245416    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f4e8547b-a373-4e35-95fa-fe3844ae2ff6" containerName="maas-api"
Nov 01 14:50:47.245489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245425    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e8547b-a373-4e35-95fa-fe3844ae2ff6" containerName="maas-api"
Nov 01 14:50:47.245489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245449    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9e834dc1-573f-4c1a-b252-898425b81872" containerName="payload-processing"
Nov 01 14:50:47.245489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245456    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e834dc1-573f-4c1a-b252-898425b81872" containerName="payload-processing"
Nov 01 14:50:47.245740 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245616    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="079a6430-8709-4bf3-a0b2-ff8489cf56a3" containerName="payload-pre-processing"
Nov 01 14:50:47.245740 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245632    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e8547b-a373-4e35-95fa-fe3844ae2ff6" containerName="maas-api"
Nov 01 14:50:47.245740 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.245643    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e834dc1-573f-4c1a-b252-898425b81872" containerName="payload-processing"
Nov 01 14:50:47.250709 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.250686    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.301189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.301161    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"]
Nov 01 14:50:47.344928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.344896    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.345077 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.345036    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da0232-0e8c-4b65-aa90-4011d051e10a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.345077 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.345061    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-metrics-certs\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.345157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.345092    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxs66\" (UniqueName: \"kubernetes.io/projected/e2da0232-0e8c-4b65-aa90-4011d051e10a-kube-api-access-fxs66\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.349955 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.349929    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-60547a-serving-cert\""
Nov 01 14:50:47.476406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.466968    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.476406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.467210    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da0232-0e8c-4b65-aa90-4011d051e10a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.476406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.467248    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-metrics-certs\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.476406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.467294    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxs66\" (UniqueName: \"kubernetes.io/projected/e2da0232-0e8c-4b65-aa90-4011d051e10a-kube-api-access-fxs66\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.476406 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:47.467351    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-serving-cert: secret "maas-api-e2e-ipp-b-60547a-serving-cert" not found
Nov 01 14:50:47.476406 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:47.467451    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls podName:e2da0232-0e8c-4b65-aa90-4011d051e10a nodeName:}" failed. No retries permitted until 2026-11-01 14:50:47.967429566 +0000 UTC m=+1988.132914845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls") pod "maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh" (UID: "e2da0232-0e8c-4b65-aa90-4011d051e10a") : secret "maas-api-e2e-ipp-b-60547a-serving-cert" not found
Nov 01 14:50:47.476406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.468328    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da0232-0e8c-4b65-aa90-4011d051e10a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.479800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.478819    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-metrics-certs\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.525990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.525921    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxs66\" (UniqueName: \"kubernetes.io/projected/e2da0232-0e8c-4b65-aa90-4011d051e10a-kube-api-access-fxs66\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.971573 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.971536    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:47.974075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:47.974047    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:48.165022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:48.164980    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:48.547369 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:50:48.547334    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2da0232_0e8c_4b65_aa90_4011d051e10a.slice/crio-db9f06e24b28d25302c785641823c0e104d76997e9907c52ba1dffa3617d01cb WatchSource:0}: Error finding container db9f06e24b28d25302c785641823c0e104d76997e9907c52ba1dffa3617d01cb: Status 404 returned error can't find the container with id db9f06e24b28d25302c785641823c0e104d76997e9907c52ba1dffa3617d01cb
Nov 01 14:50:48.550820 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:48.550762    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"]
Nov 01 14:50:49.050545 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:49.050512    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh" event={"ID":"e2da0232-0e8c-4b65-aa90-4011d051e10a","Type":"ContainerStarted","Data":"db9f06e24b28d25302c785641823c0e104d76997e9907c52ba1dffa3617d01cb"}
Nov 01 14:50:50.058169 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:50.058131    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh" event={"ID":"e2da0232-0e8c-4b65-aa90-4011d051e10a","Type":"ContainerStarted","Data":"afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9"}
Nov 01 14:50:50.058579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:50.058203    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:50.218409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:50.218342    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh" podStartSLOduration=2.332461828 podStartE2EDuration="3.218326366s" totalImagesPullingTime="885.864538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:50:50.174355482 +0000 UTC m=+1990.339840781" watchObservedRunningTime="2026-11-01 14:50:50.218326366 +0000 UTC m=+1990.383811663"
Nov 01 14:50:52.242797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.242758    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"]
Nov 01 14:50:52.243309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.243020    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg" podUID="8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e" containerName="payload-processing" containerID="cri-o://291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92" gracePeriod=30
Nov 01 14:50:52.293172 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.293142    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"]
Nov 01 14:50:52.293351 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.293332    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq" podUID="7130638f-3a21-4c18-9df9-23260b113eb3" containerName="payload-pre-processing" containerID="cri-o://b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded" gracePeriod=30
Nov 01 14:50:52.560816 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.560786    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:50:52.589061 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.589036    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:50:52.662429 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.662372    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-kube-api-access-fxh59\" (UniqueName: \"kubernetes.io/projected/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-kube-api-access-fxh59\") pod \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\" (UID: \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\") "
Nov 01 14:50:52.662581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.662437    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-plugins-config-volume\") pod \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\" (UID: \"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e\") "
Nov 01 14:50:52.662581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.662483    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7130638f-3a21-4c18-9df9-23260b113eb3-kube-api-access-dlqjr\" (UniqueName: \"kubernetes.io/projected/7130638f-3a21-4c18-9df9-23260b113eb3-kube-api-access-dlqjr\") pod \"7130638f-3a21-4c18-9df9-23260b113eb3\" (UID: \"7130638f-3a21-4c18-9df9-23260b113eb3\") "
Nov 01 14:50:52.662581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.662576    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7130638f-3a21-4c18-9df9-23260b113eb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7130638f-3a21-4c18-9df9-23260b113eb3-plugins-config-volume\") pod \"7130638f-3a21-4c18-9df9-23260b113eb3\" (UID: \"7130638f-3a21-4c18-9df9-23260b113eb3\") "
Nov 01 14:50:52.662901 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.662868    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-plugins-config-volume" pod "8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e" (UID: "8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:52.663047 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.663004    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7130638f-3a21-4c18-9df9-23260b113eb3-plugins-config-volume" pod "7130638f-3a21-4c18-9df9-23260b113eb3" (UID: "7130638f-3a21-4c18-9df9-23260b113eb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:52.664559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.664539    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-kube-api-access-fxh59" pod "8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e" (UID: "8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e"). InnerVolumeSpecName "kube-api-access-fxh59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:52.664668 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.664651    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7130638f-3a21-4c18-9df9-23260b113eb3-kube-api-access-dlqjr" pod "7130638f-3a21-4c18-9df9-23260b113eb3" (UID: "7130638f-3a21-4c18-9df9-23260b113eb3"). InnerVolumeSpecName "kube-api-access-dlqjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:52.764024 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.763989    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxh59\" (UniqueName: \"kubernetes.io/projected/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-kube-api-access-fxh59\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:52.764024 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.764018    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:52.764024 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.764029    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqjr\" (UniqueName: \"kubernetes.io/projected/7130638f-3a21-4c18-9df9-23260b113eb3-kube-api-access-dlqjr\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:52.764263 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:52.764037    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7130638f-3a21-4c18-9df9-23260b113eb3-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:53.095067 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.095029    5423 generic.go:356] "Generic (PLEG): container finished" podID="8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e" containerID="291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92" exitCode=0
Nov 01 14:50:53.095235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.095091    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"
Nov 01 14:50:53.095235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.095120    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg" event={"ID":"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e","Type":"ContainerDied","Data":"291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92"}
Nov 01 14:50:53.095235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.095190    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg" event={"ID":"8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e","Type":"ContainerDied","Data":"4ddbd84917f78794595e90ab90c1c22f04d526c7126f9262a3a0db8e00072cdc"}
Nov 01 14:50:53.095235 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.095214    5423 scope.go:122] "RemoveContainer" containerID="291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92"
Nov 01 14:50:53.096557 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.096534    5423 generic.go:356] "Generic (PLEG): container finished" podID="7130638f-3a21-4c18-9df9-23260b113eb3" containerID="b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded" exitCode=0
Nov 01 14:50:53.096684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.096591    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"
Nov 01 14:50:53.096684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.096599    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq" event={"ID":"7130638f-3a21-4c18-9df9-23260b113eb3","Type":"ContainerDied","Data":"b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded"}
Nov 01 14:50:53.096684 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.096641    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq" event={"ID":"7130638f-3a21-4c18-9df9-23260b113eb3","Type":"ContainerDied","Data":"93b797ee0b99fe8e7cb2448457bc99050d256f08107440572d4b8817488f3eed"}
Nov 01 14:50:53.105835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.105818    5423 scope.go:122] "RemoveContainer" containerID="291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92"
Nov 01 14:50:53.106116 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:53.106093    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92\": container with ID starting with 291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92 not found: ID does not exist" containerID="291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92"
Nov 01 14:50:53.106199 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.106126    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92"} err="failed to get container status \"291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92\": rpc error: code = NotFound desc = could not find container \"291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92\": container with ID starting with 291914ec811e43750759f5abe7380e5cf679541889b0016feb58c0b65b176c92 not found: ID does not exist"
Nov 01 14:50:53.106199 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.106149    5423 scope.go:122] "RemoveContainer" containerID="b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded"
Nov 01 14:50:53.115223 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.115204    5423 scope.go:122] "RemoveContainer" containerID="b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded"
Nov 01 14:50:53.115498 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:50:53.115479    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded\": container with ID starting with b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded not found: ID does not exist" containerID="b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded"
Nov 01 14:50:53.115567 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.115508    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded"} err="failed to get container status \"b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded\": rpc error: code = NotFound desc = could not find container \"b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded\": container with ID starting with b5e1d9917eb546b18049fac49584366601f5471a917f21ea4acf9bd5ac065ded not found: ID does not exist"
Nov 01 14:50:53.262775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.262752    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"]
Nov 01 14:50:53.265258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.265234    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be7e2e-69fd8cd6f5-lgknq"]
Nov 01 14:50:53.375475 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.375356    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"]
Nov 01 14:50:53.379742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.379711    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be7e2e-57d48f6888-g2ntg"]
Nov 01 14:50:53.982253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.982213    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7130638f-3a21-4c18-9df9-23260b113eb3" path="/var/lib/kubelet/pods/7130638f-3a21-4c18-9df9-23260b113eb3/volumes"
Nov 01 14:50:53.983166 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:53.983143    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e" path="/var/lib/kubelet/pods/8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e/volumes"
Nov 01 14:50:56.068687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:56.068658    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:50:58.949623 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:58.949588    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"]
Nov 01 14:50:58.950355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:58.950335    5423 cpu_manager.go:414] "containerMap: removing container" podUID="8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e" containerName="payload-processing"
Nov 01 14:50:58.950476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:58.950356    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e" containerName="payload-processing"
Nov 01 14:50:58.950476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:58.950411    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7130638f-3a21-4c18-9df9-23260b113eb3" containerName="payload-pre-processing"
Nov 01 14:50:58.950476 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:58.950420    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7130638f-3a21-4c18-9df9-23260b113eb3" containerName="payload-pre-processing"
Nov 01 14:50:58.950641 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:58.950579    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f773d43-c6b9-4da9-bf6e-0ea67c64ad7e" containerName="payload-processing"
Nov 01 14:50:58.950641 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:58.950597    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7130638f-3a21-4c18-9df9-23260b113eb3" containerName="payload-pre-processing"
Nov 01 14:50:58.955432 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:58.955408    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:50:59.043428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.035545    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"]
Nov 01 14:50:59.055709 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.055682    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0bb4b4-dockercfg-tzzmj\""
Nov 01 14:50:59.079415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.062570    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0bb4b4\""
Nov 01 14:50:59.079415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.063259    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1657784a-740d-4722-be8e-e4797da58db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4\" (UID: \"1657784a-740d-4722-be8e-e4797da58db9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:50:59.079415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.066895    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngslh\" (UniqueName: \"kubernetes.io/projected/1657784a-740d-4722-be8e-e4797da58db9-kube-api-access-ngslh\") pod \"payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4\" (UID: \"1657784a-740d-4722-be8e-e4797da58db9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:50:59.129239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.129197    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"]
Nov 01 14:50:59.135439 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.134159    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:50:59.168326 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.168301    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmxk\" (UniqueName: \"kubernetes.io/projected/1d45a52f-8646-42da-b748-aa7474f27df6-kube-api-access-gsmxk\") pod \"payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf\" (UID: \"1d45a52f-8646-42da-b748-aa7474f27df6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:50:59.168497 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.168452    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngslh\" (UniqueName: \"kubernetes.io/projected/1657784a-740d-4722-be8e-e4797da58db9-kube-api-access-ngslh\") pod \"payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4\" (UID: \"1657784a-740d-4722-be8e-e4797da58db9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:50:59.168551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.168502    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1657784a-740d-4722-be8e-e4797da58db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4\" (UID: \"1657784a-740d-4722-be8e-e4797da58db9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:50:59.168551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.168545    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d45a52f-8646-42da-b748-aa7474f27df6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf\" (UID: \"1d45a52f-8646-42da-b748-aa7474f27df6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:50:59.169160 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.169142    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1657784a-740d-4722-be8e-e4797da58db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4\" (UID: \"1657784a-740d-4722-be8e-e4797da58db9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:50:59.180452 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.180427    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"]
Nov 01 14:50:59.225957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.225897    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngslh\" (UniqueName: \"kubernetes.io/projected/1657784a-740d-4722-be8e-e4797da58db9-kube-api-access-ngslh\") pod \"payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4\" (UID: \"1657784a-740d-4722-be8e-e4797da58db9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:50:59.269052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.269028    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:50:59.269494 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.269472    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d45a52f-8646-42da-b748-aa7474f27df6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf\" (UID: \"1d45a52f-8646-42da-b748-aa7474f27df6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:50:59.269601 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.269568    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmxk\" (UniqueName: \"kubernetes.io/projected/1d45a52f-8646-42da-b748-aa7474f27df6-kube-api-access-gsmxk\") pod \"payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf\" (UID: \"1d45a52f-8646-42da-b748-aa7474f27df6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:50:59.270105 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.270087    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d45a52f-8646-42da-b748-aa7474f27df6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf\" (UID: \"1d45a52f-8646-42da-b748-aa7474f27df6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:50:59.329161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.329099    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmxk\" (UniqueName: \"kubernetes.io/projected/1d45a52f-8646-42da-b748-aa7474f27df6-kube-api-access-gsmxk\") pod \"payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf\" (UID: \"1d45a52f-8646-42da-b748-aa7474f27df6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:50:59.453276 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.453240    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:50:59.637592 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:50:59.637561    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1657784a_740d_4722_be8e_e4797da58db9.slice/crio-6c3fe707521746de199d5b3020d625270927e506de72b9dcb1fd8d2c16ddaa52 WatchSource:0}: Error finding container 6c3fe707521746de199d5b3020d625270927e506de72b9dcb1fd8d2c16ddaa52: Status 404 returned error can't find the container with id 6c3fe707521746de199d5b3020d625270927e506de72b9dcb1fd8d2c16ddaa52
Nov 01 14:50:59.640262 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.640235    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"]
Nov 01 14:50:59.823243 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:50:59.823218    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d45a52f_8646_42da_b748_aa7474f27df6.slice/crio-1f783a60a9d759342f3e27a0c1a3dfc23bafdc883a11dd7218473174a967328c WatchSource:0}: Error finding container 1f783a60a9d759342f3e27a0c1a3dfc23bafdc883a11dd7218473174a967328c: Status 404 returned error can't find the container with id 1f783a60a9d759342f3e27a0c1a3dfc23bafdc883a11dd7218473174a967328c
Nov 01 14:50:59.826824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:50:59.826797    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"]
Nov 01 14:51:00.149209 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.149103    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf" event={"ID":"1d45a52f-8646-42da-b748-aa7474f27df6","Type":"ContainerStarted","Data":"d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02"}
Nov 01 14:51:00.149209 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.149167    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf" event={"ID":"1d45a52f-8646-42da-b748-aa7474f27df6","Type":"ContainerStarted","Data":"1f783a60a9d759342f3e27a0c1a3dfc23bafdc883a11dd7218473174a967328c"}
Nov 01 14:51:00.149209 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.149195    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:51:00.150731 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.150698    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4" event={"ID":"1657784a-740d-4722-be8e-e4797da58db9","Type":"ContainerStarted","Data":"5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0"}
Nov 01 14:51:00.150850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.150751    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:51:00.150850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.150770    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4" event={"ID":"1657784a-740d-4722-be8e-e4797da58db9","Type":"ContainerStarted","Data":"6c3fe707521746de199d5b3020d625270927e506de72b9dcb1fd8d2c16ddaa52"}
Nov 01 14:51:00.267260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.267211    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf" podStartSLOduration=1.26719837 podStartE2EDuration="1.26719837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:00.26450661 +0000 UTC m=+2000.429991908" watchObservedRunningTime="2026-11-01 14:51:00.26719837 +0000 UTC m=+2000.432683667"
Nov 01 14:51:00.381446 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.381382    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4" podStartSLOduration=2.381369091 podStartE2EDuration="2.381369091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:00.379148454 +0000 UTC m=+2000.544633752" watchObservedRunningTime="2026-11-01 14:51:00.381369091 +0000 UTC m=+2000.546854388"
Nov 01 14:51:00.862334 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.862295    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"]
Nov 01 14:51:00.862613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:00.862585    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy" containerID="cri-o://bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c" gracePeriod=30
Nov 01 14:51:01.375139 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.375106    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"]
Nov 01 14:51:01.379400 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.379369    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.421491    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-envoy\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.421554    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-podinfo\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.421634    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbsz\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-kube-api-access-qrbsz\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.421692    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-credential-socket\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.421790    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-certs\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.421937    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a02998-1d4d-495c-8cca-d7da3eadd99c-istiod-ca-cert\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.421981    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-data\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.422027    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-socket\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.422412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.422053    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-token\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.472880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.440495    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"]
Nov 01 14:51:01.472880 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.462260    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e381b7-openshift-default-dockercfg-jfrxd\""
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563420    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a02998-1d4d-495c-8cca-d7da3eadd99c-istiod-ca-cert\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563549    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-data\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563609    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-socket\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563642    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-token\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563694    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-envoy\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563727    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-podinfo\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563798    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbsz\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-kube-api-access-qrbsz\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563847    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-credential-socket\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.564697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.563927    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-certs\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.593745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.591467    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-certs\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.594091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.594064    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-credential-socket\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.594547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.594319    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-data\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.594666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.594575    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-socket\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.596902 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.596877    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-envoy\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.597903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.597870    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a02998-1d4d-495c-8cca-d7da3eadd99c-istiod-ca-cert\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.598013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.597914    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-podinfo\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.637038 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.637002    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-token\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.644900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.644876    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbsz\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-kube-api-access-qrbsz\") pod \"e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") " pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:01.717901 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:01.717670    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:02.139858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:51:02.139818    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a02998_1d4d_495c_8cca_d7da3eadd99c.slice/crio-835ff9ea6b13f6b5b9b0f36b23553d6b079c52aa21af8996458ef47be8c50784 WatchSource:0}: Error finding container 835ff9ea6b13f6b5b9b0f36b23553d6b079c52aa21af8996458ef47be8c50784: Status 404 returned error can't find the container with id 835ff9ea6b13f6b5b9b0f36b23553d6b079c52aa21af8996458ef47be8c50784
Nov 01 14:51:02.142485 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:02.142443    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:51:02.142568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:02.142527    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:51:02.142604 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:02.142568    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:51:02.143987 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:02.143965    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"]
Nov 01 14:51:02.166794 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:02.166767    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" event={"ID":"35a02998-1d4d-495c-8cca-d7da3eadd99c","Type":"ContainerStarted","Data":"835ff9ea6b13f6b5b9b0f36b23553d6b079c52aa21af8996458ef47be8c50784"}
Nov 01 14:51:03.186463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:03.186426    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" event={"ID":"35a02998-1d4d-495c-8cca-d7da3eadd99c","Type":"ContainerStarted","Data":"59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7"}
Nov 01 14:51:03.361408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:03.361336    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" podStartSLOduration=2.361321307 podStartE2EDuration="2.361321307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:03.35910754 +0000 UTC m=+2003.524592841" watchObservedRunningTime="2026-11-01 14:51:03.361321307 +0000 UTC m=+2003.526806605"
Nov 01 14:51:03.717837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:03.717799    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:03.724214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:03.724178    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:04.192788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:04.192757    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:04.193664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:04.193645    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:51:05.150559 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:51:05.150524    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:51:06.131042 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.130972    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:51:06.207530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.207494    5423 generic.go:356] "Generic (PLEG): container finished" podID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerID="bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c" exitCode=0
Nov 01 14:51:06.207711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.207540    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"
Nov 01 14:51:06.207711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.207575    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" event={"ID":"0b08385e-e72e-4078-9dce-f385d0ba5ad3","Type":"ContainerDied","Data":"bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c"}
Nov 01 14:51:06.207711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.207643    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv" event={"ID":"0b08385e-e72e-4078-9dce-f385d0ba5ad3","Type":"ContainerDied","Data":"711d6b9f81a9806cb468ae252f1c81ade89919a6e2155e4e57d3ff3ff3ba8171"}
Nov 01 14:51:06.207711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.207676    5423 scope.go:122] "RemoveContainer" containerID="bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c"
Nov 01 14:51:06.218096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.218076    5423 scope.go:122] "RemoveContainer" containerID="baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8"
Nov 01 14:51:06.227027 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.227010    5423 scope.go:122] "RemoveContainer" containerID="bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c"
Nov 01 14:51:06.227298 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:51:06.227280    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c\": container with ID starting with bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c not found: ID does not exist" containerID="bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c"
Nov 01 14:51:06.227343 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.227308    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c"} err="failed to get container status \"bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c\": rpc error: code = NotFound desc = could not find container \"bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c\": container with ID starting with bec5e4cb33d2859365ee719593fe404c1df1f4e67b12474b030302b789fb143c not found: ID does not exist"
Nov 01 14:51:06.227343 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.227327    5423 scope.go:122] "RemoveContainer" containerID="baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8"
Nov 01 14:51:06.227621 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:51:06.227602    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8\": container with ID starting with baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8 not found: ID does not exist" containerID="baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8"
Nov 01 14:51:06.227682 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.227630    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8"} err="failed to get container status \"baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8\": rpc error: code = NotFound desc = could not find container \"baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8\": container with ID starting with baf6be866ff384a450ca15e509d2eaedecedd80bf9ca15e1c4f95a5904ce05c8 not found: ID does not exist"
Nov 01 14:51:06.232023 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232003    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-kube-api-access-2cbts\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-kube-api-access-2cbts\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232110 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232041    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-token\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232110 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232091    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-socket\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232205 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232122    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-data\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232266 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232250    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-credential-socket\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232325 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232292    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istiod-ca-cert\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232380 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232367    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-envoy\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232397    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-data" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:06.232464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232413    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-socket" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:06.232464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232434    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-podinfo\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232545    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-certs\") pod \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\" (UID: \"0b08385e-e72e-4078-9dce-f385d0ba5ad3\") "
Nov 01 14:51:06.232627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232536    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-credential-socket" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:06.232766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232722    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istiod-ca-cert" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:51:06.233297 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.232928    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-certs" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:06.233734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.233688    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.233734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.233722    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.233734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.233738    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.234603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.233753    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.234603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.233778    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.234603 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.234510    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-token" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:51:06.234723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.234629    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-kube-api-access-2cbts" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "kube-api-access-2cbts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:51:06.234829 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.234808    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-envoy" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:06.235215 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.235197    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-podinfo" pod "0b08385e-e72e-4078-9dce-f385d0ba5ad3" (UID: "0b08385e-e72e-4078-9dce-f385d0ba5ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:51:06.334928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.334885    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.334928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.334927    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.335162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.334946    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbts\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-kube-api-access-2cbts\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.335162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.334964    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b08385e-e72e-4078-9dce-f385d0ba5ad3-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:06.634564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.634534    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"]
Nov 01 14:51:06.636707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:06.636687    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-be7e2e-openshift-default-6c9c8c95f7-csvlv"]
Nov 01 14:51:07.961120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:07.961090    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" path="/var/lib/kubelet/pods/0b08385e-e72e-4078-9dce-f385d0ba5ad3/volumes"
Nov 01 14:51:09.473612 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.473580    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"]
Nov 01 14:51:09.474171 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.474158    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy"
Nov 01 14:51:09.474229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.474173    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy"
Nov 01 14:51:09.474229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.474190    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy"
Nov 01 14:51:09.474229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.474195    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy"
Nov 01 14:51:09.474352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.474280    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy"
Nov 01 14:51:09.474352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.474291    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b08385e-e72e-4078-9dce-f385d0ba5ad3" containerName="istio-proxy"
Nov 01 14:51:09.479239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.479218    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:09.512068 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.512039    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbdea91-7c2f-4be4-9718-03ad625fd878-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2\" (UID: \"efbdea91-7c2f-4be4-9718-03ad625fd878\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:09.512232 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.512209    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbj5w\" (UniqueName: \"kubernetes.io/projected/efbdea91-7c2f-4be4-9718-03ad625fd878-kube-api-access-lbj5w\") pod \"payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2\" (UID: \"efbdea91-7c2f-4be4-9718-03ad625fd878\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:09.605062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.603760    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e381b7-dockercfg-6qsgw\""
Nov 01 14:51:09.605062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.603861    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e381b7\""
Nov 01 14:51:09.612762 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.612711    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"]
Nov 01 14:51:09.614258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.614231    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbdea91-7c2f-4be4-9718-03ad625fd878-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2\" (UID: \"efbdea91-7c2f-4be4-9718-03ad625fd878\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:09.614436 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.613370    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbdea91-7c2f-4be4-9718-03ad625fd878-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2\" (UID: \"efbdea91-7c2f-4be4-9718-03ad625fd878\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:09.614688 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.614672    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbj5w\" (UniqueName: \"kubernetes.io/projected/efbdea91-7c2f-4be4-9718-03ad625fd878-kube-api-access-lbj5w\") pod \"payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2\" (UID: \"efbdea91-7c2f-4be4-9718-03ad625fd878\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:09.660236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.659939    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"]
Nov 01 14:51:09.664685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.664656    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:09.672914 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.672894    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbj5w\" (UniqueName: \"kubernetes.io/projected/efbdea91-7c2f-4be4-9718-03ad625fd878-kube-api-access-lbj5w\") pod \"payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2\" (UID: \"efbdea91-7c2f-4be4-9718-03ad625fd878\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:09.709936 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.709912    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"]
Nov 01 14:51:09.720100 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.720065    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mtz\" (UniqueName: \"kubernetes.io/projected/df112af2-6a75-47a5-9327-a6b6416b253c-kube-api-access-z9mtz\") pod \"payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w\" (UID: \"df112af2-6a75-47a5-9327-a6b6416b253c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:09.720226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.720190    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df112af2-6a75-47a5-9327-a6b6416b253c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w\" (UID: \"df112af2-6a75-47a5-9327-a6b6416b253c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:09.793583 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.793509    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:09.821868 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.821838    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mtz\" (UniqueName: \"kubernetes.io/projected/df112af2-6a75-47a5-9327-a6b6416b253c-kube-api-access-z9mtz\") pod \"payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w\" (UID: \"df112af2-6a75-47a5-9327-a6b6416b253c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:09.821998 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.821934    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df112af2-6a75-47a5-9327-a6b6416b253c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w\" (UID: \"df112af2-6a75-47a5-9327-a6b6416b253c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:09.822864 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.822837    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df112af2-6a75-47a5-9327-a6b6416b253c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w\" (UID: \"df112af2-6a75-47a5-9327-a6b6416b253c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:09.879656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.879628    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mtz\" (UniqueName: \"kubernetes.io/projected/df112af2-6a75-47a5-9327-a6b6416b253c-kube-api-access-z9mtz\") pod \"payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w\" (UID: \"df112af2-6a75-47a5-9327-a6b6416b253c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:09.991468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:09.991436    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:10.160723 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:51:10.160691    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbdea91_7c2f_4be4_9718_03ad625fd878.slice/crio-474a847322a95ac604317ab1799d268550c7a8e6b87fa344e96c7261449de33c WatchSource:0}: Error finding container 474a847322a95ac604317ab1799d268550c7a8e6b87fa344e96c7261449de33c: Status 404 returned error can't find the container with id 474a847322a95ac604317ab1799d268550c7a8e6b87fa344e96c7261449de33c
Nov 01 14:51:10.163575 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:10.163551    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"]
Nov 01 14:51:10.274304 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:10.274273    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2" event={"ID":"efbdea91-7c2f-4be4-9718-03ad625fd878","Type":"ContainerStarted","Data":"f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee"}
Nov 01 14:51:10.274463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:10.274325    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2" event={"ID":"efbdea91-7c2f-4be4-9718-03ad625fd878","Type":"ContainerStarted","Data":"474a847322a95ac604317ab1799d268550c7a8e6b87fa344e96c7261449de33c"}
Nov 01 14:51:10.274463 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:10.274408    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:10.358462 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:51:10.358432    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf112af2_6a75_47a5_9327_a6b6416b253c.slice/crio-ddf5fbdb73ab4c8bb5117e049ff797bf2b05e3f4047274fe9e4a6949a0d5a941 WatchSource:0}: Error finding container ddf5fbdb73ab4c8bb5117e049ff797bf2b05e3f4047274fe9e4a6949a0d5a941: Status 404 returned error can't find the container with id ddf5fbdb73ab4c8bb5117e049ff797bf2b05e3f4047274fe9e4a6949a0d5a941
Nov 01 14:51:10.361668 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:10.361646    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"]
Nov 01 14:51:10.391718 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:10.391677    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2" podStartSLOduration=1.391663023 podStartE2EDuration="1.391663023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:10.388634655 +0000 UTC m=+2010.554119952" watchObservedRunningTime="2026-11-01 14:51:10.391663023 +0000 UTC m=+2010.557148320"
Nov 01 14:51:11.160117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:11.160086    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:51:11.160642 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:11.160163    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:51:11.281953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:11.281911    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w" event={"ID":"df112af2-6a75-47a5-9327-a6b6416b253c","Type":"ContainerStarted","Data":"e73de031aac9739de4bff1a8bdf48e449b78e13f772009436fecb07a60b0ec93"}
Nov 01 14:51:11.282133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:11.281966    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w" event={"ID":"df112af2-6a75-47a5-9327-a6b6416b253c","Type":"ContainerStarted","Data":"ddf5fbdb73ab4c8bb5117e049ff797bf2b05e3f4047274fe9e4a6949a0d5a941"}
Nov 01 14:51:11.282133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:11.281993    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:12.845284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.845233    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w" podStartSLOduration=3.845220718 podStartE2EDuration="3.845220718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:11.528485169 +0000 UTC m=+2011.693970468" watchObservedRunningTime="2026-11-01 14:51:12.845220718 +0000 UTC m=+2013.010706061"
Nov 01 14:51:12.847981 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.847955    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"]
Nov 01 14:51:12.852198 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.852183    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:12.905189 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.905160    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"]
Nov 01 14:51:12.952350 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.952324    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2048c1-serving-cert\""
Nov 01 14:51:12.970170 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.970141    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9t2l\" (UniqueName: \"kubernetes.io/projected/b6dcd1d6-f381-4556-ba66-18197a1648a6-kube-api-access-p9t2l\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:12.970326 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.970214    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:12.970419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.970336    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dcd1d6-f381-4556-ba66-18197a1648a6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:12.970490 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:12.970466    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-metrics-certs\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.080260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.076118    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9t2l\" (UniqueName: \"kubernetes.io/projected/b6dcd1d6-f381-4556-ba66-18197a1648a6-kube-api-access-p9t2l\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.080260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.076213    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.080260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.076281    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dcd1d6-f381-4556-ba66-18197a1648a6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.080260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.076344    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-metrics-certs\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.080260 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:51:13.077023    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-serving-cert: secret "maas-api-e2e-ipp-a-2048c1-serving-cert" not found
Nov 01 14:51:13.080260 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:51:13.077095    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls podName:b6dcd1d6-f381-4556-ba66-18197a1648a6 nodeName:}" failed. No retries permitted until 2026-11-01 14:51:13.577074603 +0000 UTC m=+2013.742559897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls") pod "maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml" (UID: "b6dcd1d6-f381-4556-ba66-18197a1648a6") : secret "maas-api-e2e-ipp-a-2048c1-serving-cert" not found
Nov 01 14:51:13.080260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.078460    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dcd1d6-f381-4556-ba66-18197a1648a6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.119492 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.112244    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-metrics-certs\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.136033 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.136003    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9t2l\" (UniqueName: \"kubernetes.io/projected/b6dcd1d6-f381-4556-ba66-18197a1648a6-kube-api-access-p9t2l\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.582666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.582630    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.585484 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.585458    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:13.788409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:13.787738    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:14.192936 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.192834    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"]
Nov 01 14:51:14.217006 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:51:14.216949    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6dcd1d6_f381_4556_ba66_18197a1648a6.slice/crio-e94af8632d53d6cd79262a43a603a40d158d1f4e14a6975e4cd7045fe526c086 WatchSource:0}: Error finding container e94af8632d53d6cd79262a43a603a40d158d1f4e14a6975e4cd7045fe526c086: Status 404 returned error can't find the container with id e94af8632d53d6cd79262a43a603a40d158d1f4e14a6975e4cd7045fe526c086
Nov 01 14:51:14.229872 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.229846    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"]
Nov 01 14:51:14.235861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.235827    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:14.243500 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.243475    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"]
Nov 01 14:51:14.289095 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.289065    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f764c7\""
Nov 01 14:51:14.289233 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.289171    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f764c7-dockercfg-swzkm\""
Nov 01 14:51:14.301357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.301329    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml" event={"ID":"b6dcd1d6-f381-4556-ba66-18197a1648a6","Type":"ContainerStarted","Data":"e94af8632d53d6cd79262a43a603a40d158d1f4e14a6975e4cd7045fe526c086"}
Nov 01 14:51:14.404464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.404431    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8cl2\" (UniqueName: \"kubernetes.io/projected/cbed19b3-8946-4aa8-81b0-8d5874180d70-kube-api-access-f8cl2\") pod \"payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz\" (UID: \"cbed19b3-8946-4aa8-81b0-8d5874180d70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:14.404596 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.404508    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbed19b3-8946-4aa8-81b0-8d5874180d70-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz\" (UID: \"cbed19b3-8946-4aa8-81b0-8d5874180d70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:14.505541 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.505476    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cl2\" (UniqueName: \"kubernetes.io/projected/cbed19b3-8946-4aa8-81b0-8d5874180d70-kube-api-access-f8cl2\") pod \"payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz\" (UID: \"cbed19b3-8946-4aa8-81b0-8d5874180d70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:14.505643 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.505540    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbed19b3-8946-4aa8-81b0-8d5874180d70-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz\" (UID: \"cbed19b3-8946-4aa8-81b0-8d5874180d70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:14.506220 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.506198    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbed19b3-8946-4aa8-81b0-8d5874180d70-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz\" (UID: \"cbed19b3-8946-4aa8-81b0-8d5874180d70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:14.564236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.564208    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8cl2\" (UniqueName: \"kubernetes.io/projected/cbed19b3-8946-4aa8-81b0-8d5874180d70-kube-api-access-f8cl2\") pod \"payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz\" (UID: \"cbed19b3-8946-4aa8-81b0-8d5874180d70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:14.853997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:14.853968    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:15.224027 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:51:15.224001    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbed19b3_8946_4aa8_81b0_8d5874180d70.slice/crio-5c8fff4a4156f36c1b73e890bd0736e2525313836eec8cd7a7df948c6b730df3 WatchSource:0}: Error finding container 5c8fff4a4156f36c1b73e890bd0736e2525313836eec8cd7a7df948c6b730df3: Status 404 returned error can't find the container with id 5c8fff4a4156f36c1b73e890bd0736e2525313836eec8cd7a7df948c6b730df3
Nov 01 14:51:15.227336 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:15.227311    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"]
Nov 01 14:51:15.309732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:15.309688    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml" event={"ID":"b6dcd1d6-f381-4556-ba66-18197a1648a6","Type":"ContainerStarted","Data":"f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163"}
Nov 01 14:51:15.309856 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:15.309783    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:15.311418 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:15.311373    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz" event={"ID":"cbed19b3-8946-4aa8-81b0-8d5874180d70","Type":"ContainerStarted","Data":"ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a"}
Nov 01 14:51:15.311516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:15.311432    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz" event={"ID":"cbed19b3-8946-4aa8-81b0-8d5874180d70","Type":"ContainerStarted","Data":"5c8fff4a4156f36c1b73e890bd0736e2525313836eec8cd7a7df948c6b730df3"}
Nov 01 14:51:15.311592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:15.311524    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:15.426186 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:15.426096    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml" podStartSLOduration=2.60003144 podStartE2EDuration="3.426082202s" totalImagesPullingTime="826.050762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:15.425304257 +0000 UTC m=+2015.590789596" watchObservedRunningTime="2026-11-01 14:51:15.426082202 +0000 UTC m=+2015.591567500"
Nov 01 14:51:15.549606 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:15.549195    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz" podStartSLOduration=1.549155995 podStartE2EDuration="1.549155995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:15.5491438 +0000 UTC m=+2015.714629100" watchObservedRunningTime="2026-11-01 14:51:15.549155995 +0000 UTC m=+2015.714641284"
Nov 01 14:51:21.285964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:21.285933    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:51:21.324823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:21.324793    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:51:22.291528 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:22.291500    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:51:26.322310 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:26.322274    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:51:34.774669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:34.774630    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:51:35.148992 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:51:35.148957    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:51:41.138207 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.138170    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"]
Nov 01 14:51:41.143695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.143673    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.188577 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.188546    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.188719 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.188600    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-metrics-certs\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.188774 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.188713    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459e28fd-ec01-4a69-9491-55cad7526a3f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.188832 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.188783    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sss6h\" (UniqueName: \"kubernetes.io/projected/459e28fd-ec01-4a69-9491-55cad7526a3f-kube-api-access-sss6h\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.195365 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.195337    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"]
Nov 01 14:51:41.241700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.241664    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-0bb4b4-serving-cert\""
Nov 01 14:51:41.289779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.289748    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.289943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.289797    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-metrics-certs\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.289943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.289837    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459e28fd-ec01-4a69-9491-55cad7526a3f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.289943 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:51:41.289884    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-serving-cert: secret "maas-api-e2e-praxis-a-0bb4b4-serving-cert" not found
Nov 01 14:51:41.289943 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.289884    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sss6h\" (UniqueName: \"kubernetes.io/projected/459e28fd-ec01-4a69-9491-55cad7526a3f-kube-api-access-sss6h\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.290165 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:51:41.289949    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls podName:459e28fd-ec01-4a69-9491-55cad7526a3f nodeName:}" failed. No retries permitted until 2026-11-01 14:51:41.789927854 +0000 UTC m=+2041.955413145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls") pod "maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb" (UID: "459e28fd-ec01-4a69-9491-55cad7526a3f") : secret "maas-api-e2e-praxis-a-0bb4b4-serving-cert" not found
Nov 01 14:51:41.290594 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.290572    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459e28fd-ec01-4a69-9491-55cad7526a3f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.292420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.292398    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-metrics-certs\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.350823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.350792    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sss6h\" (UniqueName: \"kubernetes.io/projected/459e28fd-ec01-4a69-9491-55cad7526a3f-kube-api-access-sss6h\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.796876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.796786    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:41.799372 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:41.799347    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:42.058795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:42.058764    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:42.427237 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:51:42.427206    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459e28fd_ec01_4a69_9491_55cad7526a3f.slice/crio-fe952d6f13841d7d575f4d78228157a4ee8b6c490a197ea6b473a1780338bc4c WatchSource:0}: Error finding container fe952d6f13841d7d575f4d78228157a4ee8b6c490a197ea6b473a1780338bc4c: Status 404 returned error can't find the container with id fe952d6f13841d7d575f4d78228157a4ee8b6c490a197ea6b473a1780338bc4c
Nov 01 14:51:42.431471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:42.431438    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"]
Nov 01 14:51:42.477444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:42.477417    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb" event={"ID":"459e28fd-ec01-4a69-9491-55cad7526a3f","Type":"ContainerStarted","Data":"fe952d6f13841d7d575f4d78228157a4ee8b6c490a197ea6b473a1780338bc4c"}
Nov 01 14:51:43.487107 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:43.487070    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb" event={"ID":"459e28fd-ec01-4a69-9491-55cad7526a3f","Type":"ContainerStarted","Data":"2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490"}
Nov 01 14:51:43.487571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:43.487131    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:51:43.609775 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:43.609721    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb" podStartSLOduration=1.7780215099999999 podStartE2EDuration="2.609706824s" totalImagesPullingTime="831.685314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:43.608071011 +0000 UTC m=+2043.773556310" watchObservedRunningTime="2026-11-01 14:51:43.609706824 +0000 UTC m=+2043.775192121"
Nov 01 14:51:48.560571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:48.560542    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wc2s5_0162fda0-7a65-4c47-8aa7-46dd184f8cb5/payload-processing/0.log"
Nov 01 14:51:48.807206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:48.807179    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wc2s5_0162fda0-7a65-4c47-8aa7-46dd184f8cb5/payload-processing/0.log"
Nov 01 14:51:49.499495 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:51:49.499467    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:52:04.277952 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.277920    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"]
Nov 01 14:52:04.282170 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.282146    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.328170 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.328143    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"]
Nov 01 14:52:04.376653 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.376625    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f764c7-serving-cert\""
Nov 01 14:52:04.414512 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.414475    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-metrics-certs\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.414708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.414616    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197dcbfb-9678-4806-bee1-4d57480b02f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.414708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.414672    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wx2t\" (UniqueName: \"kubernetes.io/projected/197dcbfb-9678-4806-bee1-4d57480b02f6-kube-api-access-9wx2t\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.414835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.414711    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.535801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.530456    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-metrics-certs\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.535801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.531289    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197dcbfb-9678-4806-bee1-4d57480b02f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.535801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.531351    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wx2t\" (UniqueName: \"kubernetes.io/projected/197dcbfb-9678-4806-bee1-4d57480b02f6-kube-api-access-9wx2t\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.535801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.532330    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197dcbfb-9678-4806-bee1-4d57480b02f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.535801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.532459    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.545370 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.545332    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-metrics-certs\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.545884 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.545854    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.590724 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.590691    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wx2t\" (UniqueName: \"kubernetes.io/projected/197dcbfb-9678-4806-bee1-4d57480b02f6-kube-api-access-9wx2t\") pod \"maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:04.599962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:04.599934    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:05.009269 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:52:05.009238    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197dcbfb_9678_4806_bee1_4d57480b02f6.slice/crio-7fddd891a73b3dd754282756e9cfddee88e3aa3340efffd59528a241e2ae6e0d WatchSource:0}: Error finding container 7fddd891a73b3dd754282756e9cfddee88e3aa3340efffd59528a241e2ae6e0d: Status 404 returned error can't find the container with id 7fddd891a73b3dd754282756e9cfddee88e3aa3340efffd59528a241e2ae6e0d
Nov 01 14:52:05.013130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:05.013110    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"]
Nov 01 14:52:05.149940 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:52:05.149907    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:52:05.643939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:05.643901    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv" event={"ID":"197dcbfb-9678-4806-bee1-4d57480b02f6","Type":"ContainerStarted","Data":"7fddd891a73b3dd754282756e9cfddee88e3aa3340efffd59528a241e2ae6e0d"}
Nov 01 14:52:06.651160 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:06.651118    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv" event={"ID":"197dcbfb-9678-4806-bee1-4d57480b02f6","Type":"ContainerStarted","Data":"49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791"}
Nov 01 14:52:06.651622 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:06.651322    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:06.769814 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:06.769764    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv" podStartSLOduration=1.8069989020000001 podStartE2EDuration="2.769749774s" totalImagesPullingTime="962.750872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:06.76721457 +0000 UTC m=+2066.932699867" watchObservedRunningTime="2026-11-01 14:52:06.769749774 +0000 UTC m=+2066.935235073"
Nov 01 14:52:12.663080 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:12.663049    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:52:17.755248 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:17.755212    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"]
Nov 01 14:52:17.756035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:17.755987    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p" podUID="7b0397dc-1006-4eff-a02b-9635a774f80a" containerName="payload-processing" containerID="cri-o://3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82" gracePeriod=30
Nov 01 14:52:17.809855 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:17.809828    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"]
Nov 01 14:52:17.810038 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:17.810018    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc" podUID="9605f980-140d-40a6-aaa5-45c36e8a05e9" containerName="payload-pre-processing" containerID="cri-o://a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af" gracePeriod=30
Nov 01 14:52:18.108742 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.108719    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:52:18.111963 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.111945    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:52:18.215177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.215144    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9605f980-140d-40a6-aaa5-45c36e8a05e9-kube-api-access-6tcpz\" (UniqueName: \"kubernetes.io/projected/9605f980-140d-40a6-aaa5-45c36e8a05e9-kube-api-access-6tcpz\") pod \"9605f980-140d-40a6-aaa5-45c36e8a05e9\" (UID: \"9605f980-140d-40a6-aaa5-45c36e8a05e9\") "
Nov 01 14:52:18.215337 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.215209    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0397dc-1006-4eff-a02b-9635a774f80a-kube-api-access-c8mwn\" (UniqueName: \"kubernetes.io/projected/7b0397dc-1006-4eff-a02b-9635a774f80a-kube-api-access-c8mwn\") pod \"7b0397dc-1006-4eff-a02b-9635a774f80a\" (UID: \"7b0397dc-1006-4eff-a02b-9635a774f80a\") "
Nov 01 14:52:18.215337 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.215327    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9605f980-140d-40a6-aaa5-45c36e8a05e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9605f980-140d-40a6-aaa5-45c36e8a05e9-plugins-config-volume\") pod \"9605f980-140d-40a6-aaa5-45c36e8a05e9\" (UID: \"9605f980-140d-40a6-aaa5-45c36e8a05e9\") "
Nov 01 14:52:18.215446 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.215380    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0397dc-1006-4eff-a02b-9635a774f80a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0397dc-1006-4eff-a02b-9635a774f80a-plugins-config-volume\") pod \"7b0397dc-1006-4eff-a02b-9635a774f80a\" (UID: \"7b0397dc-1006-4eff-a02b-9635a774f80a\") "
Nov 01 14:52:18.215759 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.215728    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9605f980-140d-40a6-aaa5-45c36e8a05e9-plugins-config-volume" pod "9605f980-140d-40a6-aaa5-45c36e8a05e9" (UID: "9605f980-140d-40a6-aaa5-45c36e8a05e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:52:18.215862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.215808    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0397dc-1006-4eff-a02b-9635a774f80a-plugins-config-volume" pod "7b0397dc-1006-4eff-a02b-9635a774f80a" (UID: "7b0397dc-1006-4eff-a02b-9635a774f80a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:52:18.216021 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.216004    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9605f980-140d-40a6-aaa5-45c36e8a05e9-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.216091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.216025    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0397dc-1006-4eff-a02b-9635a774f80a-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.217424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.217367    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0397dc-1006-4eff-a02b-9635a774f80a-kube-api-access-c8mwn" pod "7b0397dc-1006-4eff-a02b-9635a774f80a" (UID: "7b0397dc-1006-4eff-a02b-9635a774f80a"). InnerVolumeSpecName "kube-api-access-c8mwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:18.217424 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.217411    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9605f980-140d-40a6-aaa5-45c36e8a05e9-kube-api-access-6tcpz" pod "9605f980-140d-40a6-aaa5-45c36e8a05e9" (UID: "9605f980-140d-40a6-aaa5-45c36e8a05e9"). InnerVolumeSpecName "kube-api-access-6tcpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:18.344425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.343349    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tcpz\" (UniqueName: \"kubernetes.io/projected/9605f980-140d-40a6-aaa5-45c36e8a05e9-kube-api-access-6tcpz\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.344425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.343419    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8mwn\" (UniqueName: \"kubernetes.io/projected/7b0397dc-1006-4eff-a02b-9635a774f80a-kube-api-access-c8mwn\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.766870 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.766789    5423 generic.go:356] "Generic (PLEG): container finished" podID="9605f980-140d-40a6-aaa5-45c36e8a05e9" containerID="a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af" exitCode=0
Nov 01 14:52:18.767342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.766893    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"
Nov 01 14:52:18.767342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.766919    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc" event={"ID":"9605f980-140d-40a6-aaa5-45c36e8a05e9","Type":"ContainerDied","Data":"a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af"}
Nov 01 14:52:18.767342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.766977    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc" event={"ID":"9605f980-140d-40a6-aaa5-45c36e8a05e9","Type":"ContainerDied","Data":"badb40fb5e6e7cfe51e160fd6b1d4a291bb32ea1524ced9feb7ef1930568ddf5"}
Nov 01 14:52:18.767342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.767001    5423 scope.go:122] "RemoveContainer" containerID="a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af"
Nov 01 14:52:18.768847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.768824    5423 generic.go:356] "Generic (PLEG): container finished" podID="7b0397dc-1006-4eff-a02b-9635a774f80a" containerID="3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82" exitCode=0
Nov 01 14:52:18.768965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.768891    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"
Nov 01 14:52:18.768965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.768908    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p" event={"ID":"7b0397dc-1006-4eff-a02b-9635a774f80a","Type":"ContainerDied","Data":"3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82"}
Nov 01 14:52:18.769066 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.768970    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p" event={"ID":"7b0397dc-1006-4eff-a02b-9635a774f80a","Type":"ContainerDied","Data":"39c607358f25c46f605b60c4a5da94ca5f0b51326e8fd5d2aa72439319083029"}
Nov 01 14:52:18.784179 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.784159    5423 scope.go:122] "RemoveContainer" containerID="a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af"
Nov 01 14:52:18.784512 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:52:18.784493    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af\": container with ID starting with a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af not found: ID does not exist" containerID="a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af"
Nov 01 14:52:18.784573 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.784521    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af"} err="failed to get container status \"a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af\": rpc error: code = NotFound desc = could not find container \"a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af\": container with ID starting with a80a8e10477a68ed53a493b808843b204aa40656eb44edd9cb2d794ea450b9af not found: ID does not exist"
Nov 01 14:52:18.784573 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.784539    5423 scope.go:122] "RemoveContainer" containerID="3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82"
Nov 01 14:52:18.794253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.794234    5423 scope.go:122] "RemoveContainer" containerID="3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82"
Nov 01 14:52:18.794578 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:52:18.794555    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82\": container with ID starting with 3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82 not found: ID does not exist" containerID="3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82"
Nov 01 14:52:18.794669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.794586    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82"} err="failed to get container status \"3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82\": rpc error: code = NotFound desc = could not find container \"3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82\": container with ID starting with 3beff102f0ffbbf1099dee98fa0c1f8cf93db4f31bf8da5942368fa831024e82 not found: ID does not exist"
Nov 01 14:52:18.936191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.936148    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"]
Nov 01 14:52:18.939360 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:18.939335    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-34d384-7db9879c9-rnddc"]
Nov 01 14:52:19.053976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:19.053906    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"]
Nov 01 14:52:19.058779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:19.058751    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-6558cbccdb-gcs6p"]
Nov 01 14:52:19.714546 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:19.714468    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"]
Nov 01 14:52:19.961213 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:19.961184    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0397dc-1006-4eff-a02b-9635a774f80a" path="/var/lib/kubelet/pods/7b0397dc-1006-4eff-a02b-9635a774f80a/volumes"
Nov 01 14:52:19.961688 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:19.961672    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9605f980-140d-40a6-aaa5-45c36e8a05e9" path="/var/lib/kubelet/pods/9605f980-140d-40a6-aaa5-45c36e8a05e9/volumes"
Nov 01 14:52:20.259862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:20.259821    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:52:21.280981 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:21.280940    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"]
Nov 01 14:52:21.281505 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:21.281259    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy" containerID="cri-o://7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771" gracePeriod=30
Nov 01 14:52:24.716514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:24.716450    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w" podUID="d31e1668-0499-44ad-9c64-4852f7d1a3c8" containerName="payload-pre-processing" containerID="cri-o://54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d" gracePeriod=25
Nov 01 14:52:25.067697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.067660    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:52:25.180926 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.180892    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31e1668-0499-44ad-9c64-4852f7d1a3c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1668-0499-44ad-9c64-4852f7d1a3c8-plugins-config-volume\") pod \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\" (UID: \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\") "
Nov 01 14:52:25.181120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.180946    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31e1668-0499-44ad-9c64-4852f7d1a3c8-kube-api-access-dspx4\" (UniqueName: \"kubernetes.io/projected/d31e1668-0499-44ad-9c64-4852f7d1a3c8-kube-api-access-dspx4\") pod \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\" (UID: \"d31e1668-0499-44ad-9c64-4852f7d1a3c8\") "
Nov 01 14:52:25.181352 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.181321    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31e1668-0499-44ad-9c64-4852f7d1a3c8-plugins-config-volume" pod "d31e1668-0499-44ad-9c64-4852f7d1a3c8" (UID: "d31e1668-0499-44ad-9c64-4852f7d1a3c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:52:25.181579 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.181548    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d31e1668-0499-44ad-9c64-4852f7d1a3c8-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:25.183631 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.183606    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31e1668-0499-44ad-9c64-4852f7d1a3c8-kube-api-access-dspx4" pod "d31e1668-0499-44ad-9c64-4852f7d1a3c8" (UID: "d31e1668-0499-44ad-9c64-4852f7d1a3c8"). InnerVolumeSpecName "kube-api-access-dspx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:25.282871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.282792    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dspx4\" (UniqueName: \"kubernetes.io/projected/d31e1668-0499-44ad-9c64-4852f7d1a3c8-kube-api-access-dspx4\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:25.839605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.839574    5423 generic.go:356] "Generic (PLEG): container finished" podID="d31e1668-0499-44ad-9c64-4852f7d1a3c8" containerID="54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d" exitCode=0
Nov 01 14:52:25.840026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.839625    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"
Nov 01 14:52:25.840026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.839643    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w" event={"ID":"d31e1668-0499-44ad-9c64-4852f7d1a3c8","Type":"ContainerDied","Data":"54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d"}
Nov 01 14:52:25.840026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.839709    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w" event={"ID":"d31e1668-0499-44ad-9c64-4852f7d1a3c8","Type":"ContainerDied","Data":"4cd85aae3bfa38467f4a2e011f263d3b94f2af5c8516c36e0484313a02df66c1"}
Nov 01 14:52:25.840026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.839730    5423 scope.go:122] "RemoveContainer" containerID="54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d"
Nov 01 14:52:25.850461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.850445    5423 scope.go:122] "RemoveContainer" containerID="54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d"
Nov 01 14:52:25.850796 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:52:25.850769    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d\": container with ID starting with 54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d not found: ID does not exist" containerID="54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d"
Nov 01 14:52:25.850898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:25.850801    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d"} err="failed to get container status \"54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d\": rpc error: code = NotFound desc = could not find container \"54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d\": container with ID starting with 54f806226bdccea97acaa7e77db646093459571c59b70d5ddeb5d3cc3afb029d not found: ID does not exist"
Nov 01 14:52:26.023073 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.023032    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"]
Nov 01 14:52:26.072330 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.072301    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-6cbbcccbdc-w427w"]
Nov 01 14:52:26.626022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.625981    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:52:26.725770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.725321    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istiod-ca-cert" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:52:26.726407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.726128    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istiod-ca-cert\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.726939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.726683    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-token\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-token\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.726939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.726736    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-data\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.726939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.726813    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-certs\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.726939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.726855    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-podinfo\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.726939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.726919    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-kube-api-access-t45dz\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-kube-api-access-t45dz\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.727244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.726963    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-credential-socket\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.727244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.727048    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-envoy\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.727244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.727085    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-socket\") pod \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\" (UID: \"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4\") "
Nov 01 14:52:26.727621 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.727489    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-data" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.727777 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.727700    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.727777 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.727739    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.729928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.729715    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-socket" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.729928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.729888    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-certs" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.730757 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.730729    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-credential-socket" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.732356 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.732318    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-envoy" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.736737 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.736707    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-podinfo" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:52:26.736854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.736780    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-kube-api-access-t45dz" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "kube-api-access-t45dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:26.738174 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.738136    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-token" pod "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" (UID: "f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:26.828878 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.828784    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.828878 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.828815    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.828878 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.828829    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.828878 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.828843    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t45dz\" (UniqueName: \"kubernetes.io/projected/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-kube-api-access-t45dz\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.828878 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.828854    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.828878 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.828866    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.829195 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.828882    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.848843 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.848809    5423 generic.go:356] "Generic (PLEG): container finished" podID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerID="7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771" exitCode=0
Nov 01 14:52:26.849221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.848853    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" event={"ID":"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4","Type":"ContainerDied","Data":"7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771"}
Nov 01 14:52:26.849221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.848892    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x" event={"ID":"f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4","Type":"ContainerDied","Data":"6b55c1abcb6c5519e4692016a8571363c69d10a7c32b04a9896b8f4eae02e5e1"}
Nov 01 14:52:26.849221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.848913    5423 scope.go:122] "RemoveContainer" containerID="7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771"
Nov 01 14:52:26.849221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.848999    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"
Nov 01 14:52:26.859126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.859105    5423 scope.go:122] "RemoveContainer" containerID="8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972"
Nov 01 14:52:26.867941 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.867921    5423 scope.go:122] "RemoveContainer" containerID="7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771"
Nov 01 14:52:26.868184 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:52:26.868153    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771\": container with ID starting with 7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771 not found: ID does not exist" containerID="7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771"
Nov 01 14:52:26.868267 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.868186    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771"} err="failed to get container status \"7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771\": rpc error: code = NotFound desc = could not find container \"7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771\": container with ID starting with 7debe887b3d194f67898693f45bd7943dec1aa595ba6c048849b56453103d771 not found: ID does not exist"
Nov 01 14:52:26.868267 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.868212    5423 scope.go:122] "RemoveContainer" containerID="8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972"
Nov 01 14:52:26.868455 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:52:26.868430    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972\": container with ID starting with 8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972 not found: ID does not exist" containerID="8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972"
Nov 01 14:52:26.868551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:26.868462    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972"} err="failed to get container status \"8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972\": rpc error: code = NotFound desc = could not find container \"8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972\": container with ID starting with 8736c2ccfd2dcab2291ed43e5739ffa82e4446dc7cc32afffd7fdd4c84dcb972 not found: ID does not exist"
Nov 01 14:52:27.025658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:27.025613    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"]
Nov 01 14:52:27.029473 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:27.029449    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-q2t9x"]
Nov 01 14:52:27.965059 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:27.965020    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31e1668-0499-44ad-9c64-4852f7d1a3c8" path="/var/lib/kubelet/pods/d31e1668-0499-44ad-9c64-4852f7d1a3c8/volumes"
Nov 01 14:52:27.965798 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:27.965770    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" path="/var/lib/kubelet/pods/f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4/volumes"
Nov 01 14:52:30.875673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:30.875586    5423 generic.go:356] "Generic (PLEG): container finished" podID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerID="59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7" exitCode=0
Nov 01 14:52:30.875673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:30.875659    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" event={"ID":"35a02998-1d4d-495c-8cca-d7da3eadd99c","Type":"ContainerDied","Data":"59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7"}
Nov 01 14:52:30.876132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:30.876092    5423 scope.go:122] "RemoveContainer" containerID="59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7"
Nov 01 14:52:30.879002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:30.878963    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:52:30.879141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:30.879060    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:52:30.879141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:30.879113    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:52:31.887180 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:31.887143    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" event={"ID":"35a02998-1d4d-495c-8cca-d7da3eadd99c","Type":"ContainerStarted","Data":"c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178"}
Nov 01 14:52:32.719411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:32.718689    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:52:32.720883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:32.720848    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 14:52:32.721016 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:32.720921    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 14:52:33.339827 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.339795    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"]
Nov 01 14:52:33.340224 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.339993    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh" podUID="e2da0232-0e8c-4b65-aa90-4011d051e10a" containerName="maas-api" containerID="cri-o://afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9" gracePeriod=30
Nov 01 14:52:33.668552 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.668529    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:52:33.719782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.719755    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 14:52:33.719953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.719805    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 14:52:33.768306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.747745    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2da0232-0e8c-4b65-aa90-4011d051e10a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da0232-0e8c-4b65-aa90-4011d051e10a-openshift-service-ca\") pod \"e2da0232-0e8c-4b65-aa90-4011d051e10a\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") "
Nov 01 14:52:33.768306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.747855    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-metrics-certs\") pod \"e2da0232-0e8c-4b65-aa90-4011d051e10a\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") "
Nov 01 14:52:33.768306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.748144    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2da0232-0e8c-4b65-aa90-4011d051e10a-kube-api-access-fxs66\" (UniqueName: \"kubernetes.io/projected/e2da0232-0e8c-4b65-aa90-4011d051e10a-kube-api-access-fxs66\") pod \"e2da0232-0e8c-4b65-aa90-4011d051e10a\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") "
Nov 01 14:52:33.768306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.748181    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls\") pod \"e2da0232-0e8c-4b65-aa90-4011d051e10a\" (UID: \"e2da0232-0e8c-4b65-aa90-4011d051e10a\") "
Nov 01 14:52:33.768306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.765161    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2da0232-0e8c-4b65-aa90-4011d051e10a-openshift-service-ca" pod "e2da0232-0e8c-4b65-aa90-4011d051e10a" (UID: "e2da0232-0e8c-4b65-aa90-4011d051e10a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:52:33.813410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.810549    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-metrics-certs" pod "e2da0232-0e8c-4b65-aa90-4011d051e10a" (UID: "e2da0232-0e8c-4b65-aa90-4011d051e10a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:52:33.813410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.811164    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2da0232-0e8c-4b65-aa90-4011d051e10a-kube-api-access-fxs66" pod "e2da0232-0e8c-4b65-aa90-4011d051e10a" (UID: "e2da0232-0e8c-4b65-aa90-4011d051e10a"). InnerVolumeSpecName "kube-api-access-fxs66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:33.813410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.811544    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls" pod "e2da0232-0e8c-4b65-aa90-4011d051e10a" (UID: "e2da0232-0e8c-4b65-aa90-4011d051e10a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:52:33.850951 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.850715    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxs66\" (UniqueName: \"kubernetes.io/projected/e2da0232-0e8c-4b65-aa90-4011d051e10a-kube-api-access-fxs66\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:33.850951 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.850755    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:33.850951 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.850784    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da0232-0e8c-4b65-aa90-4011d051e10a-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:33.850951 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.850799    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da0232-0e8c-4b65-aa90-4011d051e10a-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:33.962214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.962035    5423 generic.go:356] "Generic (PLEG): container finished" podID="e2da0232-0e8c-4b65-aa90-4011d051e10a" containerID="afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9" exitCode=0
Nov 01 14:52:33.962214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.962141    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"
Nov 01 14:52:33.968841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.968814    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh" event={"ID":"e2da0232-0e8c-4b65-aa90-4011d051e10a","Type":"ContainerDied","Data":"afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9"}
Nov 01 14:52:33.968947 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.968858    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh" event={"ID":"e2da0232-0e8c-4b65-aa90-4011d051e10a","Type":"ContainerDied","Data":"db9f06e24b28d25302c785641823c0e104d76997e9907c52ba1dffa3617d01cb"}
Nov 01 14:52:33.968947 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.968879    5423 scope.go:122] "RemoveContainer" containerID="afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9"
Nov 01 14:52:33.979118 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.979100    5423 scope.go:122] "RemoveContainer" containerID="afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9"
Nov 01 14:52:33.979429 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:52:33.979409    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9\": container with ID starting with afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9 not found: ID does not exist" containerID="afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9"
Nov 01 14:52:33.979513 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:33.979444    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9"} err="failed to get container status \"afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9\": rpc error: code = NotFound desc = could not find container \"afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9\": container with ID starting with afdff492171e61d489e6d9035dcc2f7b263699efc28a58c32c077754c28230a9 not found: ID does not exist"
Nov 01 14:52:34.132664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:34.132629    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"]
Nov 01 14:52:34.137405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:34.137366    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60547a-75bc67b7d4-mwxmh"]
Nov 01 14:52:34.192874 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:34.192845    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:52:34.719467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:34.718731    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 14:52:34.719467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:34.718791    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 14:52:35.148938 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:52:35.148902    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:52:35.719817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:35.719781    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 14:52:35.720242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:35.719841    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 14:52:35.960453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:35.960422    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2da0232-0e8c-4b65-aa90-4011d051e10a" path="/var/lib/kubelet/pods/e2da0232-0e8c-4b65-aa90-4011d051e10a/volumes"
Nov 01 14:52:36.719860 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:36.719824    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 14:52:36.720493 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:36.720444    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 14:52:37.723784 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:37.723749    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:52:37.724761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:37.724742    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:52:41.460599 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:41.460563    5423 scope.go:122] "RemoveContainer" containerID="6646508937ea9372a19c7aba9f351cd871d936b2041c268449686e51489984ff"
Nov 01 14:52:42.458037 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:42.458006    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:52:42.489025 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:42.489002    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:52:43.992091 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992048    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"]
Nov 01 14:52:43.992725 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992706    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7b0397dc-1006-4eff-a02b-9635a774f80a" containerName="payload-processing"
Nov 01 14:52:43.992772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992728    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0397dc-1006-4eff-a02b-9635a774f80a" containerName="payload-processing"
Nov 01 14:52:43.992772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992743    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9605f980-140d-40a6-aaa5-45c36e8a05e9" containerName="payload-pre-processing"
Nov 01 14:52:43.992772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992749    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9605f980-140d-40a6-aaa5-45c36e8a05e9" containerName="payload-pre-processing"
Nov 01 14:52:43.992772 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992770    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992775    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992784    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992790    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992802    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d31e1668-0499-44ad-9c64-4852f7d1a3c8" containerName="payload-pre-processing"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992810    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31e1668-0499-44ad-9c64-4852f7d1a3c8" containerName="payload-pre-processing"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992820    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e2da0232-0e8c-4b65-aa90-4011d051e10a" containerName="maas-api"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992825    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2da0232-0e8c-4b65-aa90-4011d051e10a" containerName="maas-api"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992909    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992917    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be8d74-4c65-4c2b-a1bb-9df37bc7abd4" containerName="istio-proxy"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992925    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31e1668-0499-44ad-9c64-4852f7d1a3c8" containerName="payload-pre-processing"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992931    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2da0232-0e8c-4b65-aa90-4011d051e10a" containerName="maas-api"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992943    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0397dc-1006-4eff-a02b-9635a774f80a" containerName="payload-processing"
Nov 01 14:52:43.992965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.992949    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9605f980-140d-40a6-aaa5-45c36e8a05e9" containerName="payload-pre-processing"
Nov 01 14:52:43.996669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:43.996651    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033297    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-data\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033374    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-token\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033450    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l4d4\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-kube-api-access-2l4d4\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033511    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-credential-socket\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033543    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-socket\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033636    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-envoy\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033691    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be8ba4-b0f7-43fe-b558-83d407323ed0-istiod-ca-cert\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033784    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-certs\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.035411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.033825    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-podinfo\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.061109 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.061075    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-305290-openshift-default-dockercfg-8tlgm\""
Nov 01 14:52:44.131467 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.122659    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"]
Nov 01 14:52:44.135493 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.135465    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be8ba4-b0f7-43fe-b558-83d407323ed0-istiod-ca-cert\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.136273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136254    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-certs\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.136492 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136467    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-podinfo\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.136600 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136582    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-data\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.136671 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136656    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-token\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.136729 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136701    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4d4\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-kube-api-access-2l4d4\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.136783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136773    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-credential-socket\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.136837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136810    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-socket\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.136906 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136885    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-envoy\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.137187 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.137168    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-certs\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.137586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.137566    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-data\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.137726 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.136194    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be8ba4-b0f7-43fe-b558-83d407323ed0-istiod-ca-cert\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.138069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.138050    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-credential-socket\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.138733 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.138714    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-socket\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.141359 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.141313    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-podinfo\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.141849 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.141819    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-envoy\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.211549 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.208269    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4d4\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-kube-api-access-2l4d4\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.224873 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.224840    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-token\") pod \"e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") " pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.309569 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.309540    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:44.783969 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.783943    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"]
Nov 01 14:52:44.784396 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.784342    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:52:44.784515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.784493    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:52:44.784604 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:44.784585    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:52:45.066984 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:45.066950    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" event={"ID":"89be8ba4-b0f7-43fe-b558-83d407323ed0","Type":"ContainerStarted","Data":"8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5"}
Nov 01 14:52:45.067470 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:45.067009    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" event={"ID":"89be8ba4-b0f7-43fe-b558-83d407323ed0","Type":"ContainerStarted","Data":"b2ce769b51a634499b9fbdee309512e9d4de232b513f8717c45f584d77253b32"}
Nov 01 14:52:45.190861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:45.190809    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" podStartSLOduration=2.19079595 podStartE2EDuration="2.19079595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:45.18831915 +0000 UTC m=+2105.353804448" watchObservedRunningTime="2026-11-01 14:52:45.19079595 +0000 UTC m=+2105.356281248"
Nov 01 14:52:45.315409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:45.311775    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:45.332319 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:45.332230    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:46.073656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:46.073620    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:46.074666 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:46.074641    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:52:46.774939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:46.774901    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:52:55.998543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:55.998513    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"]
Nov 01 14:52:56.001564 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.001547    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:56.064461 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.064428    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a90080\""
Nov 01 14:52:56.121062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.121031    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a90080-dockercfg-478mz\""
Nov 01 14:52:56.124553 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.124523    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"]
Nov 01 14:52:56.176462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.176414    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0797366-268b-49aa-adcf-ebcd2f3e0f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn\" (UID: \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:56.176801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.176567    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvrd\" (UniqueName: \"kubernetes.io/projected/d0797366-268b-49aa-adcf-ebcd2f3e0f29-kube-api-access-zhvrd\") pod \"payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn\" (UID: \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:56.180651 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.180616    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"]
Nov 01 14:52:56.185462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.185437    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:56.222625 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.222597    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"]
Nov 01 14:52:56.278011 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.277936    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvrd\" (UniqueName: \"kubernetes.io/projected/d0797366-268b-49aa-adcf-ebcd2f3e0f29-kube-api-access-zhvrd\") pod \"payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn\" (UID: \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:56.278137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.278045    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42xp\" (UniqueName: \"kubernetes.io/projected/b33b5b63-4b7f-414b-add6-fe8e491ba10a-kube-api-access-s42xp\") pod \"payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7\" (UID: \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:56.278137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.278069    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0797366-268b-49aa-adcf-ebcd2f3e0f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn\" (UID: \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:56.278137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.278100    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33b5b63-4b7f-414b-add6-fe8e491ba10a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7\" (UID: \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:56.278783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.278764    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0797366-268b-49aa-adcf-ebcd2f3e0f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn\" (UID: \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:56.336298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.336270    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvrd\" (UniqueName: \"kubernetes.io/projected/d0797366-268b-49aa-adcf-ebcd2f3e0f29-kube-api-access-zhvrd\") pod \"payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn\" (UID: \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:56.379086 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.379056    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42xp\" (UniqueName: \"kubernetes.io/projected/b33b5b63-4b7f-414b-add6-fe8e491ba10a-kube-api-access-s42xp\") pod \"payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7\" (UID: \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:56.379259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.379101    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33b5b63-4b7f-414b-add6-fe8e491ba10a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7\" (UID: \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:56.379803 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.379782    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33b5b63-4b7f-414b-add6-fe8e491ba10a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7\" (UID: \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:56.436852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.436814    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42xp\" (UniqueName: \"kubernetes.io/projected/b33b5b63-4b7f-414b-add6-fe8e491ba10a-kube-api-access-s42xp\") pod \"payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7\" (UID: \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:56.498913 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.498878    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:56.615543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.615515    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:56.864622 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:52:56.864589    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33b5b63_4b7f_414b_add6_fe8e491ba10a.slice/crio-571847c3ecc8647f9f6c7fbe91b87b7c91947d6f39307743ed8dc27989c904e8 WatchSource:0}: Error finding container 571847c3ecc8647f9f6c7fbe91b87b7c91947d6f39307743ed8dc27989c904e8: Status 404 returned error can't find the container with id 571847c3ecc8647f9f6c7fbe91b87b7c91947d6f39307743ed8dc27989c904e8
Nov 01 14:52:56.867570 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.867532    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"]
Nov 01 14:52:56.993683 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:56.993656    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"]
Nov 01 14:52:56.994073 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:52:56.994053    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0797366_268b_49aa_adcf_ebcd2f3e0f29.slice/crio-39fed56cb0c83988897e17e889a79b6ef5cccaf7a80bec8821a396dee035f95c WatchSource:0}: Error finding container 39fed56cb0c83988897e17e889a79b6ef5cccaf7a80bec8821a396dee035f95c: Status 404 returned error can't find the container with id 39fed56cb0c83988897e17e889a79b6ef5cccaf7a80bec8821a396dee035f95c
Nov 01 14:52:57.158314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:57.158223    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7" event={"ID":"b33b5b63-4b7f-414b-add6-fe8e491ba10a","Type":"ContainerStarted","Data":"6932b2a7888b4be16e5c518452c4cb4f02a5ca48d82826afc605b85e16950dff"}
Nov 01 14:52:57.158314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:57.158278    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7" event={"ID":"b33b5b63-4b7f-414b-add6-fe8e491ba10a","Type":"ContainerStarted","Data":"571847c3ecc8647f9f6c7fbe91b87b7c91947d6f39307743ed8dc27989c904e8"}
Nov 01 14:52:57.158810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:57.158379    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:52:57.159795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:57.159770    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn" event={"ID":"d0797366-268b-49aa-adcf-ebcd2f3e0f29","Type":"ContainerStarted","Data":"d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99"}
Nov 01 14:52:57.159918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:57.159810    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn" event={"ID":"d0797366-268b-49aa-adcf-ebcd2f3e0f29","Type":"ContainerStarted","Data":"39fed56cb0c83988897e17e889a79b6ef5cccaf7a80bec8821a396dee035f95c"}
Nov 01 14:52:57.159918 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:57.159836    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:52:57.275753 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:57.275698    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7" podStartSLOduration=1.275686472 podStartE2EDuration="1.275686472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:57.273534954 +0000 UTC m=+2117.439020252" watchObservedRunningTime="2026-11-01 14:52:57.275686472 +0000 UTC m=+2117.441171770"
Nov 01 14:52:57.399409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:52:57.389838    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn" podStartSLOduration=2.389817475 podStartE2EDuration="2.389817475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:57.38826353 +0000 UTC m=+2117.553748828" watchObservedRunningTime="2026-11-01 14:52:57.389817475 +0000 UTC m=+2117.555302774"
Nov 01 14:53:05.150311 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:05.150273    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:53:08.169128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:08.169097    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:53:08.169555 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:08.169170    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:53:13.119909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:13.119866    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"]
Nov 01 14:53:13.226708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:13.226674    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"]
Nov 01 14:53:14.470002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.469969    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"]
Nov 01 14:53:14.495794 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.495764    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:14.547412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.538616    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"]
Nov 01 14:53:14.579995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.579968    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2048c1\""
Nov 01 14:53:14.580165 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.579998    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2048c1-dockercfg-6755h\""
Nov 01 14:53:14.649312 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.649286    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsksm\" (UniqueName: \"kubernetes.io/projected/b7988724-6fe6-41d0-8f01-626458fe98f0-kube-api-access-zsksm\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt\" (UID: \"b7988724-6fe6-41d0-8f01-626458fe98f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:14.649520 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.649500    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7988724-6fe6-41d0-8f01-626458fe98f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt\" (UID: \"b7988724-6fe6-41d0-8f01-626458fe98f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:14.757506 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.751542    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7988724-6fe6-41d0-8f01-626458fe98f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt\" (UID: \"b7988724-6fe6-41d0-8f01-626458fe98f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:14.757883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.757855    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsksm\" (UniqueName: \"kubernetes.io/projected/b7988724-6fe6-41d0-8f01-626458fe98f0-kube-api-access-zsksm\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt\" (UID: \"b7988724-6fe6-41d0-8f01-626458fe98f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:14.767711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.756337    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7988724-6fe6-41d0-8f01-626458fe98f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt\" (UID: \"b7988724-6fe6-41d0-8f01-626458fe98f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:14.817220 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.817187    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsksm\" (UniqueName: \"kubernetes.io/projected/b7988724-6fe6-41d0-8f01-626458fe98f0-kube-api-access-zsksm\") pod \"payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt\" (UID: \"b7988724-6fe6-41d0-8f01-626458fe98f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:14.831023 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:14.831000    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:15.200079 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:53:15.200046    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7988724_6fe6_41d0_8f01_626458fe98f0.slice/crio-df3fc0eed861c743892cf596d38237d94607fedd1f9625618488bccf47550b7f WatchSource:0}: Error finding container df3fc0eed861c743892cf596d38237d94607fedd1f9625618488bccf47550b7f: Status 404 returned error can't find the container with id df3fc0eed861c743892cf596d38237d94607fedd1f9625618488bccf47550b7f
Nov 01 14:53:15.202873 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.202850    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"]
Nov 01 14:53:15.262774 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.262750    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"]
Nov 01 14:53:15.268084 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.268065    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:15.273443 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.273417    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt" event={"ID":"b7988724-6fe6-41d0-8f01-626458fe98f0","Type":"ContainerStarted","Data":"df3fc0eed861c743892cf596d38237d94607fedd1f9625618488bccf47550b7f"}
Nov 01 14:53:15.325185 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.325146    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"]
Nov 01 14:53:15.366324 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.366288    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmddr\" (UniqueName: \"kubernetes.io/projected/8819031c-9890-49b6-8677-e37a34d67d25-kube-api-access-fmddr\") pod \"payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt\" (UID: \"8819031c-9890-49b6-8677-e37a34d67d25\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:15.366526 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.366366    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8819031c-9890-49b6-8677-e37a34d67d25-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt\" (UID: \"8819031c-9890-49b6-8677-e37a34d67d25\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:15.467889 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.467814    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmddr\" (UniqueName: \"kubernetes.io/projected/8819031c-9890-49b6-8677-e37a34d67d25-kube-api-access-fmddr\") pod \"payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt\" (UID: \"8819031c-9890-49b6-8677-e37a34d67d25\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:15.468057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.467885    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8819031c-9890-49b6-8677-e37a34d67d25-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt\" (UID: \"8819031c-9890-49b6-8677-e37a34d67d25\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:15.468638 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.468617    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8819031c-9890-49b6-8677-e37a34d67d25-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt\" (UID: \"8819031c-9890-49b6-8677-e37a34d67d25\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:15.527173 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.527137    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmddr\" (UniqueName: \"kubernetes.io/projected/8819031c-9890-49b6-8677-e37a34d67d25-kube-api-access-fmddr\") pod \"payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt\" (UID: \"8819031c-9890-49b6-8677-e37a34d67d25\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:15.582550 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.582516    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:15.964643 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:53:15.964601    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8819031c_9890_49b6_8677_e37a34d67d25.slice/crio-529ec068141cc733b4d59ec1142f29136dabf350d5a04180d390ce48bc541f04 WatchSource:0}: Error finding container 529ec068141cc733b4d59ec1142f29136dabf350d5a04180d390ce48bc541f04: Status 404 returned error can't find the container with id 529ec068141cc733b4d59ec1142f29136dabf350d5a04180d390ce48bc541f04
Nov 01 14:53:15.968374 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:15.968348    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"]
Nov 01 14:53:16.281087 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:16.280993    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt" event={"ID":"8819031c-9890-49b6-8677-e37a34d67d25","Type":"ContainerStarted","Data":"bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190"}
Nov 01 14:53:16.281260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:16.281096    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:16.281260 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:16.281125    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt" event={"ID":"8819031c-9890-49b6-8677-e37a34d67d25","Type":"ContainerStarted","Data":"529ec068141cc733b4d59ec1142f29136dabf350d5a04180d390ce48bc541f04"}
Nov 01 14:53:16.282751 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:16.282722    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt" event={"ID":"b7988724-6fe6-41d0-8f01-626458fe98f0","Type":"ContainerStarted","Data":"91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d"}
Nov 01 14:53:16.282865 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:16.282818    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:16.397493 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:16.397438    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt" podStartSLOduration=1.3974218299999999 podStartE2EDuration="1.39742183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:16.396230179 +0000 UTC m=+2136.561715477" watchObservedRunningTime="2026-11-01 14:53:16.39742183 +0000 UTC m=+2136.562907128"
Nov 01 14:53:18.120629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.120587    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k" podUID="ba341519-804c-48d1-aa1c-be401139bbaf" containerName="payload-processing" containerID="cri-o://d2fa78e298e3e6ee9870889a67a5fc8b087a83a592f2de36ff87bb0502677cb5" gracePeriod=25
Nov 01 14:53:18.227852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.227818    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx" podUID="59d7fca4-25b7-4e2c-a04e-f8553a325b67" containerName="payload-pre-processing" containerID="cri-o://7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da" gracePeriod=25
Nov 01 14:53:18.297872 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.297766    5423 generic.go:356] "Generic (PLEG): container finished" podID="ba341519-804c-48d1-aa1c-be401139bbaf" containerID="d2fa78e298e3e6ee9870889a67a5fc8b087a83a592f2de36ff87bb0502677cb5" exitCode=0
Nov 01 14:53:18.298073 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.297948    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k" event={"ID":"ba341519-804c-48d1-aa1c-be401139bbaf","Type":"ContainerDied","Data":"d2fa78e298e3e6ee9870889a67a5fc8b087a83a592f2de36ff87bb0502677cb5"}
Nov 01 14:53:18.420507 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.420481    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:53:18.520436 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.513423    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba341519-804c-48d1-aa1c-be401139bbaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba341519-804c-48d1-aa1c-be401139bbaf-plugins-config-volume\") pod \"ba341519-804c-48d1-aa1c-be401139bbaf\" (UID: \"ba341519-804c-48d1-aa1c-be401139bbaf\") "
Nov 01 14:53:18.520436 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.513564    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba341519-804c-48d1-aa1c-be401139bbaf-kube-api-access-2jccs\" (UniqueName: \"kubernetes.io/projected/ba341519-804c-48d1-aa1c-be401139bbaf-kube-api-access-2jccs\") pod \"ba341519-804c-48d1-aa1c-be401139bbaf\" (UID: \"ba341519-804c-48d1-aa1c-be401139bbaf\") "
Nov 01 14:53:18.540216 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.532623    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba341519-804c-48d1-aa1c-be401139bbaf-plugins-config-volume" pod "ba341519-804c-48d1-aa1c-be401139bbaf" (UID: "ba341519-804c-48d1-aa1c-be401139bbaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:53:18.541613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.541414    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba341519-804c-48d1-aa1c-be401139bbaf-kube-api-access-2jccs" pod "ba341519-804c-48d1-aa1c-be401139bbaf" (UID: "ba341519-804c-48d1-aa1c-be401139bbaf"). InnerVolumeSpecName "kube-api-access-2jccs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:53:18.558712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.558661    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt" podStartSLOduration=4.558642998 podStartE2EDuration="4.558642998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:16.530670099 +0000 UTC m=+2136.696155398" watchObservedRunningTime="2026-11-01 14:53:18.558642998 +0000 UTC m=+2138.724128296"
Nov 01 14:53:18.571206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.571180    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:53:18.615039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.615014    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba341519-804c-48d1-aa1c-be401139bbaf-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:18.615039 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.615039    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jccs\" (UniqueName: \"kubernetes.io/projected/ba341519-804c-48d1-aa1c-be401139bbaf-kube-api-access-2jccs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:18.717270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.716787    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d7fca4-25b7-4e2c-a04e-f8553a325b67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d7fca4-25b7-4e2c-a04e-f8553a325b67-plugins-config-volume\") pod \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\" (UID: \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\") "
Nov 01 14:53:18.717270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.716906    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d7fca4-25b7-4e2c-a04e-f8553a325b67-kube-api-access-mzz5z\" (UniqueName: \"kubernetes.io/projected/59d7fca4-25b7-4e2c-a04e-f8553a325b67-kube-api-access-mzz5z\") pod \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\" (UID: \"59d7fca4-25b7-4e2c-a04e-f8553a325b67\") "
Nov 01 14:53:18.717555 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.717283    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d7fca4-25b7-4e2c-a04e-f8553a325b67-plugins-config-volume" pod "59d7fca4-25b7-4e2c-a04e-f8553a325b67" (UID: "59d7fca4-25b7-4e2c-a04e-f8553a325b67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:53:18.719413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.717693    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59d7fca4-25b7-4e2c-a04e-f8553a325b67-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:18.725298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.725264    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d7fca4-25b7-4e2c-a04e-f8553a325b67-kube-api-access-mzz5z" pod "59d7fca4-25b7-4e2c-a04e-f8553a325b67" (UID: "59d7fca4-25b7-4e2c-a04e-f8553a325b67"). InnerVolumeSpecName "kube-api-access-mzz5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:53:18.818939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:18.818899    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzz5z\" (UniqueName: \"kubernetes.io/projected/59d7fca4-25b7-4e2c-a04e-f8553a325b67-kube-api-access-mzz5z\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:19.306265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.306232    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"
Nov 01 14:53:19.306265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.306256    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k" event={"ID":"ba341519-804c-48d1-aa1c-be401139bbaf","Type":"ContainerDied","Data":"266fb0805c6344657b6d88cae0846504619e9aaa93d74b350a819c733167e441"}
Nov 01 14:53:19.306853 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.306322    5423 scope.go:122] "RemoveContainer" containerID="d2fa78e298e3e6ee9870889a67a5fc8b087a83a592f2de36ff87bb0502677cb5"
Nov 01 14:53:19.307958 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.307927    5423 generic.go:356] "Generic (PLEG): container finished" podID="59d7fca4-25b7-4e2c-a04e-f8553a325b67" containerID="7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da" exitCode=0
Nov 01 14:53:19.308071 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.307961    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx" event={"ID":"59d7fca4-25b7-4e2c-a04e-f8553a325b67","Type":"ContainerDied","Data":"7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da"}
Nov 01 14:53:19.308071 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.307978    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"
Nov 01 14:53:19.308071 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.307988    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx" event={"ID":"59d7fca4-25b7-4e2c-a04e-f8553a325b67","Type":"ContainerDied","Data":"b747c1c1bdcd3730086643dadb47648b4b3b96b1c9f2812d52f1d0a2375aad3a"}
Nov 01 14:53:19.317474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.317450    5423 scope.go:122] "RemoveContainer" containerID="7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da"
Nov 01 14:53:19.327830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.327812    5423 scope.go:122] "RemoveContainer" containerID="7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da"
Nov 01 14:53:19.328109 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:19.328083    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da\": container with ID starting with 7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da not found: ID does not exist" containerID="7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da"
Nov 01 14:53:19.328226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.328116    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da"} err="failed to get container status \"7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da\": rpc error: code = NotFound desc = could not find container \"7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da\": container with ID starting with 7120e098301e8ffcade7ab5271e12eaa145343e4a0aadf0fa52f5d181cd198da not found: ID does not exist"
Nov 01 14:53:19.473483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.473409    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"]
Nov 01 14:53:19.524691 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.524655    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d195c1-fb78bcffc-9m59k"]
Nov 01 14:53:19.591349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.591322    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"]
Nov 01 14:53:19.595332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.595308    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d195c1-75494454d7-p2cgx"]
Nov 01 14:53:19.961754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.961721    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d7fca4-25b7-4e2c-a04e-f8553a325b67" path="/var/lib/kubelet/pods/59d7fca4-25b7-4e2c-a04e-f8553a325b67/volumes"
Nov 01 14:53:19.962129 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:19.962114    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba341519-804c-48d1-aa1c-be401139bbaf" path="/var/lib/kubelet/pods/ba341519-804c-48d1-aa1c-be401139bbaf/volumes"
Nov 01 14:53:27.291317 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:27.291280    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:53:27.291783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:27.291369    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:53:35.149806 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:35.149768    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:53:38.433005 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.432964    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"]
Nov 01 14:53:38.433426 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.433188    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc" podUID="493c4113-cb63-48dc-b340-c2ca52d80cae" containerName="maas-api" containerID="cri-o://f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46" gracePeriod=30
Nov 01 14:53:38.740732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.740702    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:53:38.770034 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.770001    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493c4113-cb63-48dc-b340-c2ca52d80cae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493c4113-cb63-48dc-b340-c2ca52d80cae-openshift-service-ca\") pod \"493c4113-cb63-48dc-b340-c2ca52d80cae\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") "
Nov 01 14:53:38.770181 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.770040    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-metrics-certs\") pod \"493c4113-cb63-48dc-b340-c2ca52d80cae\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") "
Nov 01 14:53:38.770181 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.770111    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493c4113-cb63-48dc-b340-c2ca52d80cae-kube-api-access-f6zjp\" (UniqueName: \"kubernetes.io/projected/493c4113-cb63-48dc-b340-c2ca52d80cae-kube-api-access-f6zjp\") pod \"493c4113-cb63-48dc-b340-c2ca52d80cae\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") "
Nov 01 14:53:38.770286 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.770187    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls\") pod \"493c4113-cb63-48dc-b340-c2ca52d80cae\" (UID: \"493c4113-cb63-48dc-b340-c2ca52d80cae\") "
Nov 01 14:53:38.770466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.770443    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493c4113-cb63-48dc-b340-c2ca52d80cae-openshift-service-ca" pod "493c4113-cb63-48dc-b340-c2ca52d80cae" (UID: "493c4113-cb63-48dc-b340-c2ca52d80cae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:53:38.770718 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.770701    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493c4113-cb63-48dc-b340-c2ca52d80cae-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:38.772291 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.772266    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls" pod "493c4113-cb63-48dc-b340-c2ca52d80cae" (UID: "493c4113-cb63-48dc-b340-c2ca52d80cae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:53:38.772397 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.772341    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-metrics-certs" pod "493c4113-cb63-48dc-b340-c2ca52d80cae" (UID: "493c4113-cb63-48dc-b340-c2ca52d80cae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:53:38.772397 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.772362    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493c4113-cb63-48dc-b340-c2ca52d80cae-kube-api-access-f6zjp" pod "493c4113-cb63-48dc-b340-c2ca52d80cae" (UID: "493c4113-cb63-48dc-b340-c2ca52d80cae"). InnerVolumeSpecName "kube-api-access-f6zjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:53:38.871932 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.871888    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:38.872098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.871938    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6zjp\" (UniqueName: \"kubernetes.io/projected/493c4113-cb63-48dc-b340-c2ca52d80cae-kube-api-access-f6zjp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:38.872098 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:38.871954    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493c4113-cb63-48dc-b340-c2ca52d80cae-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:39.456706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.456674    5423 generic.go:356] "Generic (PLEG): container finished" podID="493c4113-cb63-48dc-b340-c2ca52d80cae" containerID="f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46" exitCode=0
Nov 01 14:53:39.457110 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.456719    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"
Nov 01 14:53:39.457110 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.456769    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc" event={"ID":"493c4113-cb63-48dc-b340-c2ca52d80cae","Type":"ContainerDied","Data":"f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46"}
Nov 01 14:53:39.457110 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.456826    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc" event={"ID":"493c4113-cb63-48dc-b340-c2ca52d80cae","Type":"ContainerDied","Data":"957ba20e416d242c910281bae8e5eae5b58a549d363900d3eee3663bf1ca3232"}
Nov 01 14:53:39.457110 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.456851    5423 scope.go:122] "RemoveContainer" containerID="f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46"
Nov 01 14:53:39.467368 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.467352    5423 scope.go:122] "RemoveContainer" containerID="f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46"
Nov 01 14:53:39.467642 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:39.467624    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46\": container with ID starting with f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46 not found: ID does not exist" containerID="f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46"
Nov 01 14:53:39.467705 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.467652    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46"} err="failed to get container status \"f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46\": rpc error: code = NotFound desc = could not find container \"f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46\": container with ID starting with f780392dcf5c952ebde0e1f4937ff154f022f3f4e41d5b0ab4205d93f78b4b46 not found: ID does not exist"
Nov 01 14:53:39.624794 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.624760    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"]
Nov 01 14:53:39.629634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.629605    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc"]
Nov 01 14:53:39.962876 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:39.962841    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493c4113-cb63-48dc-b340-c2ca52d80cae" path="/var/lib/kubelet/pods/493c4113-cb63-48dc-b340-c2ca52d80cae/volumes"
Nov 01 14:53:41.723076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:41.723029    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d195c1-64645b79bc-dllvc" podUID="493c4113-cb63-48dc-b340-c2ca52d80cae" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.130:8443/health\": context deadline exceeded"
Nov 01 14:53:43.394959 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:43.394911    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:53:44.742313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:44.742279    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"]
Nov 01 14:53:44.743032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:44.742512    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy" containerID="cri-o://37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103" gracePeriod=30
Nov 01 14:53:49.489134 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489051    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"]
Nov 01 14:53:49.489652 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489637    5423 cpu_manager.go:414] "containerMap: removing container" podUID="59d7fca4-25b7-4e2c-a04e-f8553a325b67" containerName="payload-pre-processing"
Nov 01 14:53:49.489695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489652    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d7fca4-25b7-4e2c-a04e-f8553a325b67" containerName="payload-pre-processing"
Nov 01 14:53:49.489695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489677    5423 cpu_manager.go:414] "containerMap: removing container" podUID="ba341519-804c-48d1-aa1c-be401139bbaf" containerName="payload-processing"
Nov 01 14:53:49.489695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489682    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba341519-804c-48d1-aa1c-be401139bbaf" containerName="payload-processing"
Nov 01 14:53:49.489795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489697    5423 cpu_manager.go:414] "containerMap: removing container" podUID="493c4113-cb63-48dc-b340-c2ca52d80cae" containerName="maas-api"
Nov 01 14:53:49.489795 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489702    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493c4113-cb63-48dc-b340-c2ca52d80cae" containerName="maas-api"
Nov 01 14:53:49.489861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489801    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="493c4113-cb63-48dc-b340-c2ca52d80cae" containerName="maas-api"
Nov 01 14:53:49.489861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489815    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d7fca4-25b7-4e2c-a04e-f8553a325b67" containerName="payload-pre-processing"
Nov 01 14:53:49.489861 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.489824    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba341519-804c-48d1-aa1c-be401139bbaf" containerName="payload-processing"
Nov 01 14:53:49.494916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.494894    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.523788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.523757    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12926b0f-5040-45fa-b1cd-b99c4e784c2b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.523920 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.523800    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hc8\" (UniqueName: \"kubernetes.io/projected/12926b0f-5040-45fa-b1cd-b99c4e784c2b-kube-api-access-w4hc8\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.523920 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.523820    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.524004 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.523949    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-metrics-certs\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.545530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.545500    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"]
Nov 01 14:53:49.594468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.594432    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a90080-serving-cert\""
Nov 01 14:53:49.625903 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.625862    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12926b0f-5040-45fa-b1cd-b99c4e784c2b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.626070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.625927    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hc8\" (UniqueName: \"kubernetes.io/projected/12926b0f-5040-45fa-b1cd-b99c4e784c2b-kube-api-access-w4hc8\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.626070 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.625959    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.626203 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:49.626091    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-serving-cert: secret "maas-api-e2e-praxis-a-a90080-serving-cert" not found
Nov 01 14:53:49.626203 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.626147    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-metrics-certs\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.626203 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:49.626160    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls podName:12926b0f-5040-45fa-b1cd-b99c4e784c2b nodeName:}" failed. No retries permitted until 2026-11-01 14:53:50.126140345 +0000 UTC m=+2170.291625644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls") pod "maas-api-e2e-praxis-a-a90080-6db754c75b-htd29" (UID: "12926b0f-5040-45fa-b1cd-b99c4e784c2b") : secret "maas-api-e2e-praxis-a-a90080-serving-cert" not found
Nov 01 14:53:49.626658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.626636    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12926b0f-5040-45fa-b1cd-b99c4e784c2b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.629284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.629260    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-metrics-certs\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:49.686438 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:49.686381    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hc8\" (UniqueName: \"kubernetes.io/projected/12926b0f-5040-45fa-b1cd-b99c4e784c2b-kube-api-access-w4hc8\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:50.049779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.049755    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:53:50.133738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.133710    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-token\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-token\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.133909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.133819    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-kube-api-access-x647j\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-kube-api-access-x647j\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.133909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.133859    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-certs\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.134028 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.133918    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-socket\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.134028 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.133964    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec6d420-5b16-4a0b-868c-819eea8a7368-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec6d420-5b16-4a0b-868c-819eea8a7368-istiod-ca-cert\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.134028 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.133997    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-podinfo\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.134191 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134100    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-credential-socket\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.134247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134188    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-envoy\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.134247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134183    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-socket" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:53:50.134247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134218    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-data\") pod \"8ec6d420-5b16-4a0b-868c-819eea8a7368\" (UID: \"8ec6d420-5b16-4a0b-868c-819eea8a7368\") "
Nov 01 14:53:50.134445 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134432    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-credential-socket" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:53:50.134511 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134192    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-certs" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:53:50.134511 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134486    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-data" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:53:50.134584 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134529    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec6d420-5b16-4a0b-868c-819eea8a7368-istiod-ca-cert" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:53:50.134673 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134647    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:50.134822 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134800    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.134822 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134821    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.135076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134836    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec6d420-5b16-4a0b-868c-819eea8a7368-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.135076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134845    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.135076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.134854    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.136229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.136204    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-kube-api-access-x647j" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "kube-api-access-x647j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:53:50.136337 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.136318    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-token" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:53:50.136520 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.136496    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-envoy" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:53:50.136620 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.136598    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-podinfo" pod "8ec6d420-5b16-4a0b-868c-819eea8a7368" (UID: "8ec6d420-5b16-4a0b-868c-819eea8a7368"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:53:50.137036 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.137018    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a90080-6db754c75b-htd29\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:50.236010 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.235972    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.236010 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.236002    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.236198 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.236018    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x647j\" (UniqueName: \"kubernetes.io/projected/8ec6d420-5b16-4a0b-868c-819eea8a7368-kube-api-access-x647j\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.236198 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.236031    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec6d420-5b16-4a0b-868c-819eea8a7368-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:50.408960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.408929    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:50.547802 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.547771    5423 generic.go:356] "Generic (PLEG): container finished" podID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerID="37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103" exitCode=0
Nov 01 14:53:50.548211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.547813    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" event={"ID":"8ec6d420-5b16-4a0b-868c-819eea8a7368","Type":"ContainerDied","Data":"37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103"}
Nov 01 14:53:50.548211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.547834    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"
Nov 01 14:53:50.548211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.547876    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p" event={"ID":"8ec6d420-5b16-4a0b-868c-819eea8a7368","Type":"ContainerDied","Data":"a9e6356df948a3cdc1ce3f5bf0e32339151c6a728ef02f25c536c409edfa9440"}
Nov 01 14:53:50.548211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.547898    5423 scope.go:122] "RemoveContainer" containerID="37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103"
Nov 01 14:53:50.557525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.557509    5423 scope.go:122] "RemoveContainer" containerID="a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f"
Nov 01 14:53:50.565972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.565956    5423 scope.go:122] "RemoveContainer" containerID="37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103"
Nov 01 14:53:50.566230 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:50.566211    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103\": container with ID starting with 37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103 not found: ID does not exist" containerID="37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103"
Nov 01 14:53:50.566309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.566243    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103"} err="failed to get container status \"37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103\": rpc error: code = NotFound desc = could not find container \"37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103\": container with ID starting with 37c60cd282da544de2166173097588d7d12fd2f2d7fb3ac06692e8fb2d623103 not found: ID does not exist"
Nov 01 14:53:50.566309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.566268    5423 scope.go:122] "RemoveContainer" containerID="a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f"
Nov 01 14:53:50.566561 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:50.566543    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f\": container with ID starting with a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f not found: ID does not exist" containerID="a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f"
Nov 01 14:53:50.566607 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.566568    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f"} err="failed to get container status \"a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f\": rpc error: code = NotFound desc = could not find container \"a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f\": container with ID starting with a549ad283b5e01a59d298fe176ea104524128db375ad4e701d5b50fad67ed86f not found: ID does not exist"
Nov 01 14:53:50.715802 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.715749    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"]
Nov 01 14:53:50.766639 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.766610    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d195c1-openshift-default-74fbd65b85-7zz5p"]
Nov 01 14:53:50.858518 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:53:50.858491    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12926b0f_5040_45fa_b1cd_b99c4e784c2b.slice/crio-48455a4e9dfdc9da604a3e941cec4fa53a433a5ee36c7d5a505c93d77a2ed099 WatchSource:0}: Error finding container 48455a4e9dfdc9da604a3e941cec4fa53a433a5ee36c7d5a505c93d77a2ed099: Status 404 returned error can't find the container with id 48455a4e9dfdc9da604a3e941cec4fa53a433a5ee36c7d5a505c93d77a2ed099
Nov 01 14:53:50.861693 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:50.861672    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"]
Nov 01 14:53:51.197887 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.197851    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"]
Nov 01 14:53:51.198513 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.198497    5423 cpu_manager.go:414] "containerMap: removing container" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy"
Nov 01 14:53:51.198571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.198515    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy"
Nov 01 14:53:51.198629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.198618    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy"
Nov 01 14:53:51.203557 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.203534    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.247938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.247894    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-podinfo\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.248127 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.248056    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istiod-ca-cert\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.248127 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.248107    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-certs\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.248253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.248177    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-envoy\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.248403 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.248372    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-credential-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.248492 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.248438    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-token\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.248546 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.248519    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4rw5\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-kube-api-access-q4rw5\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.248628 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.248609    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-data\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.248691 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.248648    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.256848 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.256825    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-288f1e-openshift-default-dockercfg-lthh4\""
Nov 01 14:53:51.257357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.257330    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"]
Nov 01 14:53:51.349464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349436    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istiod-ca-cert\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.349628 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349482    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-certs\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.349628 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349599    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-envoy\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.349738 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349697    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-credential-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.349801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349759    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-token\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.349801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349790    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4rw5\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-kube-api-access-q4rw5\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.349908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349831    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-data\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.349908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349841    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-certs\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.349908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349879    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.350072 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.349946    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-podinfo\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.350216 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.350192    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istiod-ca-cert\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.350293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.350224    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-credential-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.350351 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.350325    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-data\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.350428 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.350377    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.352115 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.352085    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-envoy\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.352736 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.352704    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-podinfo\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.407598 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.407572    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-token\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.407740 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.407654    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4rw5\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-kube-api-access-q4rw5\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.518697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.518616    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:51.557050 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.557015    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29" event={"ID":"12926b0f-5040-45fa-b1cd-b99c4e784c2b","Type":"ContainerStarted","Data":"48455a4e9dfdc9da604a3e941cec4fa53a433a5ee36c7d5a505c93d77a2ed099"}
Nov 01 14:53:51.946423 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:53:51.944547    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fea37e_0d2a_46ac_8c9d_cbe688baa79c.slice/crio-eaacd69933924f9b3fa82e47d6646afd457d6e76c30ac4cd30bc9ee1b0e29be5 WatchSource:0}: Error finding container eaacd69933924f9b3fa82e47d6646afd457d6e76c30ac4cd30bc9ee1b0e29be5: Status 404 returned error can't find the container with id eaacd69933924f9b3fa82e47d6646afd457d6e76c30ac4cd30bc9ee1b0e29be5
Nov 01 14:53:51.947851 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.947822    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"]
Nov 01 14:53:51.947977 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.947948    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:53:51.948038 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.948009    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:53:51.948096 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.948039    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:53:51.961909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:51.961883    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" path="/var/lib/kubelet/pods/8ec6d420-5b16-4a0b-868c-819eea8a7368/volumes"
Nov 01 14:53:52.586868 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:52.586830    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" event={"ID":"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c","Type":"ContainerStarted","Data":"f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e"}
Nov 01 14:53:52.587365 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:52.586884    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" event={"ID":"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c","Type":"ContainerStarted","Data":"eaacd69933924f9b3fa82e47d6646afd457d6e76c30ac4cd30bc9ee1b0e29be5"}
Nov 01 14:53:52.593420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:52.589851    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29" event={"ID":"12926b0f-5040-45fa-b1cd-b99c4e784c2b","Type":"ContainerStarted","Data":"d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e"}
Nov 01 14:53:52.593420 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:52.590573    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:52.725243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:52.725171    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" podStartSLOduration=1.725151582 podStartE2EDuration="1.725151582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:52.713875324 +0000 UTC m=+2172.879360624" watchObservedRunningTime="2026-11-01 14:53:52.725151582 +0000 UTC m=+2172.890636881"
Nov 01 14:53:52.834466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:52.834379    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29" podStartSLOduration=2.904122848 podStartE2EDuration="3.834363617s" totalImagesPullingTime="930.240769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:52.833592963 +0000 UTC m=+2172.999078263" watchObservedRunningTime="2026-11-01 14:53:52.834363617 +0000 UTC m=+2172.999848914"
Nov 01 14:53:53.519605 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:53.519568    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:53.521893 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:53.521854    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 14:53:53.522036 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:53.521918    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 14:53:53.524268 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:53.524247    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"]
Nov 01 14:53:53.630809 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:53.630771    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"]
Nov 01 14:53:54.519898 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:54.519856    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 14:53:54.520093 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:54.519930    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 14:53:55.519789 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:55.519745    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 14:53:55.520264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:55.519814    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 14:53:56.519782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:56.519738    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 14:53:56.519980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:56.519808    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 14:53:57.519831 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:57.519784    5423 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 01 14:53:57.520087 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:57.519866    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 01 14:53:58.523886 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.523855    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:58.524302 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.524249    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:58.524536 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.524506    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w" podUID="df112af2-6a75-47a5-9327-a6b6416b253c" containerName="payload-processing" containerID="cri-o://e73de031aac9739de4bff1a8bdf48e449b78e13f772009436fecb07a60b0ec93" gracePeriod=25
Nov 01 14:53:58.525174 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.525072    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:53:58.631200 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.631168    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2" podUID="efbdea91-7c2f-4be4-9718-03ad625fd878" containerName="payload-pre-processing" containerID="cri-o://f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee" gracePeriod=25
Nov 01 14:53:58.664677 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.664645    5423 generic.go:356] "Generic (PLEG): container finished" podID="df112af2-6a75-47a5-9327-a6b6416b253c" containerID="e73de031aac9739de4bff1a8bdf48e449b78e13f772009436fecb07a60b0ec93" exitCode=0
Nov 01 14:53:58.664842 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.664725    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w" event={"ID":"df112af2-6a75-47a5-9327-a6b6416b253c","Type":"ContainerDied","Data":"e73de031aac9739de4bff1a8bdf48e449b78e13f772009436fecb07a60b0ec93"}
Nov 01 14:53:58.796896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.796865    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:53:58.910298 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.910274    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df112af2-6a75-47a5-9327-a6b6416b253c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df112af2-6a75-47a5-9327-a6b6416b253c-plugins-config-volume\") pod \"df112af2-6a75-47a5-9327-a6b6416b253c\" (UID: \"df112af2-6a75-47a5-9327-a6b6416b253c\") "
Nov 01 14:53:58.910444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.910353    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df112af2-6a75-47a5-9327-a6b6416b253c-kube-api-access-z9mtz\" (UniqueName: \"kubernetes.io/projected/df112af2-6a75-47a5-9327-a6b6416b253c-kube-api-access-z9mtz\") pod \"df112af2-6a75-47a5-9327-a6b6416b253c\" (UID: \"df112af2-6a75-47a5-9327-a6b6416b253c\") "
Nov 01 14:53:58.910687 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.910664    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df112af2-6a75-47a5-9327-a6b6416b253c-plugins-config-volume" pod "df112af2-6a75-47a5-9327-a6b6416b253c" (UID: "df112af2-6a75-47a5-9327-a6b6416b253c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:53:58.912221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.912201    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df112af2-6a75-47a5-9327-a6b6416b253c-kube-api-access-z9mtz" pod "df112af2-6a75-47a5-9327-a6b6416b253c" (UID: "df112af2-6a75-47a5-9327-a6b6416b253c"). InnerVolumeSpecName "kube-api-access-z9mtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:53:58.917313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:58.917292    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:53:59.011966 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.011930    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbdea91-7c2f-4be4-9718-03ad625fd878-kube-api-access-lbj5w\" (UniqueName: \"kubernetes.io/projected/efbdea91-7c2f-4be4-9718-03ad625fd878-kube-api-access-lbj5w\") pod \"efbdea91-7c2f-4be4-9718-03ad625fd878\" (UID: \"efbdea91-7c2f-4be4-9718-03ad625fd878\") "
Nov 01 14:53:59.012128 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.012116    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbdea91-7c2f-4be4-9718-03ad625fd878-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbdea91-7c2f-4be4-9718-03ad625fd878-plugins-config-volume\") pod \"efbdea91-7c2f-4be4-9718-03ad625fd878\" (UID: \"efbdea91-7c2f-4be4-9718-03ad625fd878\") "
Nov 01 14:53:59.012495 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.012472    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbdea91-7c2f-4be4-9718-03ad625fd878-plugins-config-volume" pod "efbdea91-7c2f-4be4-9718-03ad625fd878" (UID: "efbdea91-7c2f-4be4-9718-03ad625fd878"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:53:59.012629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.012613    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df112af2-6a75-47a5-9327-a6b6416b253c-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:59.012669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.012632    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbdea91-7c2f-4be4-9718-03ad625fd878-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:59.012669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.012642    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mtz\" (UniqueName: \"kubernetes.io/projected/df112af2-6a75-47a5-9327-a6b6416b253c-kube-api-access-z9mtz\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:59.014135 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.014113    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbdea91-7c2f-4be4-9718-03ad625fd878-kube-api-access-lbj5w" pod "efbdea91-7c2f-4be4-9718-03ad625fd878" (UID: "efbdea91-7c2f-4be4-9718-03ad625fd878"). InnerVolumeSpecName "kube-api-access-lbj5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:53:59.114273 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.114234    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbj5w\" (UniqueName: \"kubernetes.io/projected/efbdea91-7c2f-4be4-9718-03ad625fd878-kube-api-access-lbj5w\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:59.636482 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.636452    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:53:59.671892 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.671860    5423 generic.go:356] "Generic (PLEG): container finished" podID="efbdea91-7c2f-4be4-9718-03ad625fd878" containerID="f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee" exitCode=0
Nov 01 14:53:59.672012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.671902    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"
Nov 01 14:53:59.672012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.671944    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2" event={"ID":"efbdea91-7c2f-4be4-9718-03ad625fd878","Type":"ContainerDied","Data":"f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee"}
Nov 01 14:53:59.672012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.671995    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2" event={"ID":"efbdea91-7c2f-4be4-9718-03ad625fd878","Type":"ContainerDied","Data":"474a847322a95ac604317ab1799d268550c7a8e6b87fa344e96c7261449de33c"}
Nov 01 14:53:59.672140 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.672023    5423 scope.go:122] "RemoveContainer" containerID="f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee"
Nov 01 14:53:59.673658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.673627    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w" event={"ID":"df112af2-6a75-47a5-9327-a6b6416b253c","Type":"ContainerDied","Data":"ddf5fbdb73ab4c8bb5117e049ff797bf2b05e3f4047274fe9e4a6949a0d5a941"}
Nov 01 14:53:59.673809 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.673797    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"
Nov 01 14:53:59.685608 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.685586    5423 scope.go:122] "RemoveContainer" containerID="f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee"
Nov 01 14:53:59.685892 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:53:59.685870    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee\": container with ID starting with f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee not found: ID does not exist" containerID="f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee"
Nov 01 14:53:59.685981 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.685911    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee"} err="failed to get container status \"f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee\": rpc error: code = NotFound desc = could not find container \"f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee\": container with ID starting with f40aca46e57543096a4d0497a60850de963fd5c19db1b66bd7ca478ffd741aee not found: ID does not exist"
Nov 01 14:53:59.685981 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.685930    5423 scope.go:122] "RemoveContainer" containerID="e73de031aac9739de4bff1a8bdf48e449b78e13f772009436fecb07a60b0ec93"
Nov 01 14:53:59.870381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.870351    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"]
Nov 01 14:53:59.876214 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.876187    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e381b7-6547d6c5fc-cb79w"]
Nov 01 14:53:59.960332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:53:59.960305    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df112af2-6a75-47a5-9327-a6b6416b253c" path="/var/lib/kubelet/pods/df112af2-6a75-47a5-9327-a6b6416b253c/volumes"
Nov 01 14:54:00.033703 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:00.033670    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"]
Nov 01 14:54:00.047444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:00.037210    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e381b7-7997cf875-fs6l2"]
Nov 01 14:54:01.960370 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:01.960339    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbdea91-7c2f-4be4-9718-03ad625fd878" path="/var/lib/kubelet/pods/efbdea91-7c2f-4be4-9718-03ad625fd878/volumes"
Nov 01 14:54:04.949516 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:04.949484    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"]
Nov 01 14:54:04.949950 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:04.949716    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy" containerID="cri-o://c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178" gracePeriod=30
Nov 01 14:54:05.169904 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:54:05.169865    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:54:07.726996 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:07.726955    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:54:07.727414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:07.727011    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:54:10.247018 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.246987    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:54:10.399294 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399261    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-token\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-token\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399479 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399317    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-data\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399479 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399351    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-kube-api-access-qrbsz\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-kube-api-access-qrbsz\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399479 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399429    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a02998-1d4d-495c-8cca-d7da3eadd99c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a02998-1d4d-495c-8cca-d7da3eadd99c-istiod-ca-cert\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399522    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-envoy\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399554    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-socket\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399616 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399595    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-podinfo\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399642    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-certs\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399702    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-data" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:10.399766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399738    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-credential-socket\") pod \"35a02998-1d4d-495c-8cca-d7da3eadd99c\" (UID: \"35a02998-1d4d-495c-8cca-d7da3eadd99c\") "
Nov 01 14:54:10.399928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399769    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a02998-1d4d-495c-8cca-d7da3eadd99c-istiod-ca-cert" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:54:10.399928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399828    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-socket" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:10.400038 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.399944    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-certs" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:10.400126 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.400102    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-credential-socket" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:10.400349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.400327    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.400444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.400351    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.400444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.400365    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a02998-1d4d-495c-8cca-d7da3eadd99c-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.400444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.400378    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.400444 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.400418    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.401735 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.401708    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-kube-api-access-qrbsz" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "kube-api-access-qrbsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:54:10.401735 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.401723    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-token" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:54:10.402069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.402050    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-envoy" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:10.402210 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.402190    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-podinfo" pod "35a02998-1d4d-495c-8cca-d7da3eadd99c" (UID: "35a02998-1d4d-495c-8cca-d7da3eadd99c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:54:10.501522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.501450    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrbsz\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-kube-api-access-qrbsz\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.501522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.501481    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.501522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.501491    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.501522 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.501500    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a02998-1d4d-495c-8cca-d7da3eadd99c-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:10.781514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.781428    5423 generic.go:356] "Generic (PLEG): container finished" podID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerID="c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178" exitCode=0
Nov 01 14:54:10.781514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.781480    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" event={"ID":"35a02998-1d4d-495c-8cca-d7da3eadd99c","Type":"ContainerDied","Data":"c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178"}
Nov 01 14:54:10.781712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.781516    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"
Nov 01 14:54:10.781712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.781568    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp" event={"ID":"35a02998-1d4d-495c-8cca-d7da3eadd99c","Type":"ContainerDied","Data":"835ff9ea6b13f6b5b9b0f36b23553d6b079c52aa21af8996458ef47be8c50784"}
Nov 01 14:54:10.781712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.781598    5423 scope.go:122] "RemoveContainer" containerID="c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178"
Nov 01 14:54:10.792045 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.792025    5423 scope.go:122] "RemoveContainer" containerID="59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7"
Nov 01 14:54:10.801145 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.801125    5423 scope.go:122] "RemoveContainer" containerID="c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178"
Nov 01 14:54:10.801437 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:54:10.801420    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178\": container with ID starting with c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178 not found: ID does not exist" containerID="c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178"
Nov 01 14:54:10.801495 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.801444    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178"} err="failed to get container status \"c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178\": rpc error: code = NotFound desc = could not find container \"c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178\": container with ID starting with c448c2e1a64bc490dd235ba5f51bd7c2ee46775cbb357dcd2978e21a4f974178 not found: ID does not exist"
Nov 01 14:54:10.801495 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.801461    5423 scope.go:122] "RemoveContainer" containerID="59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7"
Nov 01 14:54:10.801715 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:54:10.801694    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7\": container with ID starting with 59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7 not found: ID does not exist" containerID="59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7"
Nov 01 14:54:10.801766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.801721    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7"} err="failed to get container status \"59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7\": rpc error: code = NotFound desc = could not find container \"59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7\": container with ID starting with 59ab0f4ef92de7c7a1f1036af68e41b96b1053212f294698669da829bd529fc7 not found: ID does not exist"
Nov 01 14:54:10.951702 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:10.951674    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"]
Nov 01 14:54:11.020877 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:11.018879    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e381b7-openshift-default-6b559b9b5-vgqxp"]
Nov 01 14:54:11.960489 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:11.960466    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" path="/var/lib/kubelet/pods/35a02998-1d4d-495c-8cca-d7da3eadd99c/volumes"
Nov 01 14:54:12.797472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:12.797441    5423 generic.go:356] "Generic (PLEG): container finished" podID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerID="8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5" exitCode=0
Nov 01 14:54:12.797641 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:12.797514    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" event={"ID":"89be8ba4-b0f7-43fe-b558-83d407323ed0","Type":"ContainerDied","Data":"8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5"}
Nov 01 14:54:12.797986 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:12.797970    5423 scope.go:122] "RemoveContainer" containerID="8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5"
Nov 01 14:54:12.799659 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:12.799634    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:54:12.799740 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:12.799697    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:54:12.799779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:12.799748    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:54:13.826823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:13.826771    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" event={"ID":"89be8ba4-b0f7-43fe-b558-83d407323ed0","Type":"ContainerStarted","Data":"95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7"}
Nov 01 14:54:14.315571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:14.315540    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:54:14.321015 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:14.320981    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 14:54:14.321289 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:14.321258    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 14:54:15.320140 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:15.320101    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 14:54:15.320634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:15.320166    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 14:54:16.073807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:16.073769    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:54:16.320002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:16.319966    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 14:54:16.320282 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:16.320254    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 14:54:17.062454 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.062421    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"]
Nov 01 14:54:17.063086 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063064    5423 cpu_manager.go:414] "containerMap: removing container" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy"
Nov 01 14:54:17.063086 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063085    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy"
Nov 01 14:54:17.063255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063101    5423 cpu_manager.go:414] "containerMap: removing container" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy"
Nov 01 14:54:17.063255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063108    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy"
Nov 01 14:54:17.063255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063130    5423 cpu_manager.go:414] "containerMap: removing container" podUID="df112af2-6a75-47a5-9327-a6b6416b253c" containerName="payload-processing"
Nov 01 14:54:17.063255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063137    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df112af2-6a75-47a5-9327-a6b6416b253c" containerName="payload-processing"
Nov 01 14:54:17.063255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063165    5423 cpu_manager.go:414] "containerMap: removing container" podUID="efbdea91-7c2f-4be4-9718-03ad625fd878" containerName="payload-pre-processing"
Nov 01 14:54:17.063255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063172    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbdea91-7c2f-4be4-9718-03ad625fd878" containerName="payload-pre-processing"
Nov 01 14:54:17.063255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063210    5423 cpu_manager.go:414] "containerMap: removing container" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy"
Nov 01 14:54:17.063255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063220    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy"
Nov 01 14:54:17.063704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063358    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="df112af2-6a75-47a5-9327-a6b6416b253c" containerName="payload-processing"
Nov 01 14:54:17.063704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063371    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbdea91-7c2f-4be4-9718-03ad625fd878" containerName="payload-pre-processing"
Nov 01 14:54:17.063704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063402    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec6d420-5b16-4a0b-868c-819eea8a7368" containerName="istio-proxy"
Nov 01 14:54:17.063704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063416    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy"
Nov 01 14:54:17.063704 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.063424    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a02998-1d4d-495c-8cca-d7da3eadd99c" containerName="istio-proxy"
Nov 01 14:54:17.067114 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.067094    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.089824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.089794    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r87bj\" (UniqueName: \"kubernetes.io/projected/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-kube-api-access-r87bj\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.089963 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.089839    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-metrics-certs\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.089963 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.089921    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.090074 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.089981    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.127872 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.127837    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"]
Nov 01 14:54:17.173850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.173824    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-5223a8-serving-cert\""
Nov 01 14:54:17.191258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.191228    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r87bj\" (UniqueName: \"kubernetes.io/projected/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-kube-api-access-r87bj\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.191408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.191270    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-metrics-certs\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.191408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.191316    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.191408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.191349    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.191590 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:54:17.191509    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-serving-cert: secret "maas-api-e2e-praxis-b-5223a8-serving-cert" not found
Nov 01 14:54:17.191590 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:54:17.191570    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls podName:e00ee4b7-5cf6-47a2-ad77-1c0642ba0606 nodeName:}" failed. No retries permitted until 2026-11-01 14:54:17.691554967 +0000 UTC m=+2197.857040247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls") pod "maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm" (UID: "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606") : secret "maas-api-e2e-praxis-b-5223a8-serving-cert" not found
Nov 01 14:54:17.191992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.191971    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.193799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.193778    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-metrics-certs\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.251124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.251095    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r87bj\" (UniqueName: \"kubernetes.io/projected/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-kube-api-access-r87bj\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.316590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.310706    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 14:54:17.316590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.310766    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 14:54:17.713258 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.713165    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.720311 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.720241    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:17.982405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:17.982320    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:18.320545 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:18.320509    5423 patch_prober.go:29] interesting pod/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 14:54:18.320723 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:18.320576    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 14:54:18.377535 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:54:18.377502    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00ee4b7_5cf6_47a2_ad77_1c0642ba0606.slice/crio-083f10daff55eecd6bc9ccaa4b66d61819243cb6f9a383768a62920c55129d06 WatchSource:0}: Error finding container 083f10daff55eecd6bc9ccaa4b66d61819243cb6f9a383768a62920c55129d06: Status 404 returned error can't find the container with id 083f10daff55eecd6bc9ccaa4b66d61819243cb6f9a383768a62920c55129d06
Nov 01 14:54:18.378961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:18.378935    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"]
Nov 01 14:54:18.861245 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:18.861209    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm" event={"ID":"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606","Type":"ContainerStarted","Data":"083f10daff55eecd6bc9ccaa4b66d61819243cb6f9a383768a62920c55129d06"}
Nov 01 14:54:19.315530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:19.315507    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:54:19.316541 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:19.316521    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:54:19.870333 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:19.870293    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm" event={"ID":"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606","Type":"ContainerStarted","Data":"d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067"}
Nov 01 14:54:19.870808 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:19.870449    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:19.994881 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:19.994817    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm" podStartSLOduration=2.072887074 podStartE2EDuration="2.994801346s" totalImagesPullingTime="921.914272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:54:19.992266265 +0000 UTC m=+2200.157751559" watchObservedRunningTime="2026-11-01 14:54:19.994801346 +0000 UTC m=+2200.160286645"
Nov 01 14:54:25.882363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:25.882328    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:54:31.906303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:54:31.906266    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:54:35.150902 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:54:35.150865    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:55:01.355106 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:01.355068    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:55:04.683942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:04.683906    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"]
Nov 01 14:55:04.784148 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:04.784106    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"]
Nov 01 14:55:05.149034 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:55:05.148995    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:55:09.685052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:09.684999    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf" podUID="1d45a52f-8646-42da-b748-aa7474f27df6" containerName="payload-processing" containerID="cri-o://d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02" gracePeriod=25
Nov 01 14:55:09.785295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:09.785222    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4" podUID="1657784a-740d-4722-be8e-e4797da58db9" containerName="payload-pre-processing" containerID="cri-o://5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0" gracePeriod=25
Nov 01 14:55:09.965827 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:09.965789    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:55:10.059249 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.059223    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:55:10.107447 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.107413    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d45a52f-8646-42da-b748-aa7474f27df6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d45a52f-8646-42da-b748-aa7474f27df6-plugins-config-volume\") pod \"1d45a52f-8646-42da-b748-aa7474f27df6\" (UID: \"1d45a52f-8646-42da-b748-aa7474f27df6\") "
Nov 01 14:55:10.107626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.107603    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1657784a-740d-4722-be8e-e4797da58db9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1657784a-740d-4722-be8e-e4797da58db9-plugins-config-volume\") pod \"1657784a-740d-4722-be8e-e4797da58db9\" (UID: \"1657784a-740d-4722-be8e-e4797da58db9\") "
Nov 01 14:55:10.107782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.107763    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1657784a-740d-4722-be8e-e4797da58db9-kube-api-access-ngslh\" (UniqueName: \"kubernetes.io/projected/1657784a-740d-4722-be8e-e4797da58db9-kube-api-access-ngslh\") pod \"1657784a-740d-4722-be8e-e4797da58db9\" (UID: \"1657784a-740d-4722-be8e-e4797da58db9\") "
Nov 01 14:55:10.107874 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.107810    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d45a52f-8646-42da-b748-aa7474f27df6-kube-api-access-gsmxk\" (UniqueName: \"kubernetes.io/projected/1d45a52f-8646-42da-b748-aa7474f27df6-kube-api-access-gsmxk\") pod \"1d45a52f-8646-42da-b748-aa7474f27df6\" (UID: \"1d45a52f-8646-42da-b748-aa7474f27df6\") "
Nov 01 14:55:10.107955 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.107928    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d45a52f-8646-42da-b748-aa7474f27df6-plugins-config-volume" pod "1d45a52f-8646-42da-b748-aa7474f27df6" (UID: "1d45a52f-8646-42da-b748-aa7474f27df6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:55:10.108019 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.107984    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1657784a-740d-4722-be8e-e4797da58db9-plugins-config-volume" pod "1657784a-740d-4722-be8e-e4797da58db9" (UID: "1657784a-740d-4722-be8e-e4797da58db9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:55:10.108236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.108217    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d45a52f-8646-42da-b748-aa7474f27df6-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:55:10.108293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.108237    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1657784a-740d-4722-be8e-e4797da58db9-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:55:10.109981 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.109952    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d45a52f-8646-42da-b748-aa7474f27df6-kube-api-access-gsmxk" pod "1d45a52f-8646-42da-b748-aa7474f27df6" (UID: "1d45a52f-8646-42da-b748-aa7474f27df6"). InnerVolumeSpecName "kube-api-access-gsmxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:55:10.110080 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.110010    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1657784a-740d-4722-be8e-e4797da58db9-kube-api-access-ngslh" pod "1657784a-740d-4722-be8e-e4797da58db9" (UID: "1657784a-740d-4722-be8e-e4797da58db9"). InnerVolumeSpecName "kube-api-access-ngslh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:55:10.209248 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.209164    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngslh\" (UniqueName: \"kubernetes.io/projected/1657784a-740d-4722-be8e-e4797da58db9-kube-api-access-ngslh\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:55:10.209248 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.209191    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsmxk\" (UniqueName: \"kubernetes.io/projected/1d45a52f-8646-42da-b748-aa7474f27df6-kube-api-access-gsmxk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:55:10.210023 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.209994    5423 generic.go:356] "Generic (PLEG): container finished" podID="1d45a52f-8646-42da-b748-aa7474f27df6" containerID="d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02" exitCode=0
Nov 01 14:55:10.210151 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.210106    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf" event={"ID":"1d45a52f-8646-42da-b748-aa7474f27df6","Type":"ContainerDied","Data":"d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02"}
Nov 01 14:55:10.210221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.210155    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf" event={"ID":"1d45a52f-8646-42da-b748-aa7474f27df6","Type":"ContainerDied","Data":"1f783a60a9d759342f3e27a0c1a3dfc23bafdc883a11dd7218473174a967328c"}
Nov 01 14:55:10.210221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.210202    5423 scope.go:122] "RemoveContainer" containerID="d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02"
Nov 01 14:55:10.210367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.210352    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"
Nov 01 14:55:10.218654 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.218629    5423 generic.go:356] "Generic (PLEG): container finished" podID="1657784a-740d-4722-be8e-e4797da58db9" containerID="5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0" exitCode=0
Nov 01 14:55:10.218761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.218699    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4" event={"ID":"1657784a-740d-4722-be8e-e4797da58db9","Type":"ContainerDied","Data":"5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0"}
Nov 01 14:55:10.218761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.218739    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4" event={"ID":"1657784a-740d-4722-be8e-e4797da58db9","Type":"ContainerDied","Data":"6c3fe707521746de199d5b3020d625270927e506de72b9dcb1fd8d2c16ddaa52"}
Nov 01 14:55:10.218866 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.218804    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"
Nov 01 14:55:10.226754 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.226736    5423 scope.go:122] "RemoveContainer" containerID="d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02"
Nov 01 14:55:10.227046 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:55:10.227028    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02\": container with ID starting with d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02 not found: ID does not exist" containerID="d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02"
Nov 01 14:55:10.227131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.227055    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02"} err="failed to get container status \"d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02\": rpc error: code = NotFound desc = could not find container \"d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02\": container with ID starting with d8ad19f779374cfa4f9e4c80fd368df87fe7f35514b73848b4173f5701118e02 not found: ID does not exist"
Nov 01 14:55:10.227131 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.227077    5423 scope.go:122] "RemoveContainer" containerID="5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0"
Nov 01 14:55:10.235835 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.235815    5423 scope.go:122] "RemoveContainer" containerID="5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0"
Nov 01 14:55:10.236107 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:55:10.236087    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0\": container with ID starting with 5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0 not found: ID does not exist" containerID="5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0"
Nov 01 14:55:10.236155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.236112    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0"} err="failed to get container status \"5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0\": rpc error: code = NotFound desc = could not find container \"5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0\": container with ID starting with 5e7ea6405860e3adb7b18eaddffe8769c1218495eb07236290c913a12a9501b0 not found: ID does not exist"
Nov 01 14:55:10.382821 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.382787    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"]
Nov 01 14:55:10.386120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.386096    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0bb4b4-9c8876495-vvjf4"]
Nov 01 14:55:10.532473 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.522476    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"]
Nov 01 14:55:10.534910 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:10.534856    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0bb4b4-7f6bb99c5f-dhjpf"]
Nov 01 14:55:11.960318 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:11.960283    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1657784a-740d-4722-be8e-e4797da58db9" path="/var/lib/kubelet/pods/1657784a-740d-4722-be8e-e4797da58db9/volumes"
Nov 01 14:55:11.960761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:11.960746    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d45a52f-8646-42da-b748-aa7474f27df6" path="/var/lib/kubelet/pods/1d45a52f-8646-42da-b748-aa7474f27df6/volumes"
Nov 01 14:55:32.505888 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.505806    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"]
Nov 01 14:55:32.506661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.506397    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1657784a-740d-4722-be8e-e4797da58db9" containerName="payload-pre-processing"
Nov 01 14:55:32.506661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.506410    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1657784a-740d-4722-be8e-e4797da58db9" containerName="payload-pre-processing"
Nov 01 14:55:32.506661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.506423    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1d45a52f-8646-42da-b748-aa7474f27df6" containerName="payload-processing"
Nov 01 14:55:32.506661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.506429    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d45a52f-8646-42da-b748-aa7474f27df6" containerName="payload-processing"
Nov 01 14:55:32.506661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.506533    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1657784a-740d-4722-be8e-e4797da58db9" containerName="payload-pre-processing"
Nov 01 14:55:32.506661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.506541    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d45a52f-8646-42da-b748-aa7474f27df6" containerName="payload-processing"
Nov 01 14:55:32.511231 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.511198    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:32.562533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.562504    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"]
Nov 01 14:55:32.611244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.611214    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-288f1e\""
Nov 01 14:55:32.611419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.611318    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-288f1e-dockercfg-84vrp\""
Nov 01 14:55:32.623875 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.623849    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67dfd\" (UniqueName: \"kubernetes.io/projected/b8e22105-74fd-42bc-90bc-103772edf87a-kube-api-access-67dfd\") pod \"payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w\" (UID: \"b8e22105-74fd-42bc-90bc-103772edf87a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:32.624063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.624048    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e22105-74fd-42bc-90bc-103772edf87a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w\" (UID: \"b8e22105-74fd-42bc-90bc-103772edf87a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:32.724972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.724938    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e22105-74fd-42bc-90bc-103772edf87a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w\" (UID: \"b8e22105-74fd-42bc-90bc-103772edf87a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:32.725117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.724981    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67dfd\" (UniqueName: \"kubernetes.io/projected/b8e22105-74fd-42bc-90bc-103772edf87a-kube-api-access-67dfd\") pod \"payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w\" (UID: \"b8e22105-74fd-42bc-90bc-103772edf87a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:32.725668 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.725648    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e22105-74fd-42bc-90bc-103772edf87a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w\" (UID: \"b8e22105-74fd-42bc-90bc-103772edf87a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:32.782748 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.782676    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67dfd\" (UniqueName: \"kubernetes.io/projected/b8e22105-74fd-42bc-90bc-103772edf87a-kube-api-access-67dfd\") pod \"payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w\" (UID: \"b8e22105-74fd-42bc-90bc-103772edf87a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:32.825264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.825234    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:32.954718 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.954687    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"]
Nov 01 14:55:32.959022 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:32.958944    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:33.010361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.010325    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"]
Nov 01 14:55:33.028247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.028213    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh\" (UID: \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:33.028909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.028881    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhl7h\" (UniqueName: \"kubernetes.io/projected/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-kube-api-access-hhl7h\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh\" (UID: \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:33.130092 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.130061    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhl7h\" (UniqueName: \"kubernetes.io/projected/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-kube-api-access-hhl7h\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh\" (UID: \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:33.130272 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.130149    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh\" (UID: \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:33.130844 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.130823    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh\" (UID: \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:33.188660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.188626    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhl7h\" (UniqueName: \"kubernetes.io/projected/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-kube-api-access-hhl7h\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh\" (UID: \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:33.192908 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:55:33.192883    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e22105_74fd_42bc_90bc_103772edf87a.slice/crio-bbb13d1f5a92a9bc1cbfe0b007348b7b73adf588bb35d151a01c5f5621795f81 WatchSource:0}: Error finding container bbb13d1f5a92a9bc1cbfe0b007348b7b73adf588bb35d151a01c5f5621795f81: Status 404 returned error can't find the container with id bbb13d1f5a92a9bc1cbfe0b007348b7b73adf588bb35d151a01c5f5621795f81
Nov 01 14:55:33.196818 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.196795    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"]
Nov 01 14:55:33.273700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.273673    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:33.365777 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.365736    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w" event={"ID":"b8e22105-74fd-42bc-90bc-103772edf87a","Type":"ContainerStarted","Data":"f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96"}
Nov 01 14:55:33.365956 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.365816    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:33.365956 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.365842    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w" event={"ID":"b8e22105-74fd-42bc-90bc-103772edf87a","Type":"ContainerStarted","Data":"bbb13d1f5a92a9bc1cbfe0b007348b7b73adf588bb35d151a01c5f5621795f81"}
Nov 01 14:55:33.481413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.481296    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w" podStartSLOduration=1.481284868 podStartE2EDuration="1.481284868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:33.480542104 +0000 UTC m=+2273.646027401" watchObservedRunningTime="2026-11-01 14:55:33.481284868 +0000 UTC m=+2273.646770162"
Nov 01 14:55:33.643791 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:55:33.643759    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9d5278_63b2_4148_9cfa_9ce46f31bf6f.slice/crio-4c2af58972ac4579b8c87ee78d5594d035566e7d59617d13aa365e5cd0b72415 WatchSource:0}: Error finding container 4c2af58972ac4579b8c87ee78d5594d035566e7d59617d13aa365e5cd0b72415: Status 404 returned error can't find the container with id 4c2af58972ac4579b8c87ee78d5594d035566e7d59617d13aa365e5cd0b72415
Nov 01 14:55:33.645647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:33.645623    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"]
Nov 01 14:55:34.373992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:34.373956    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh" event={"ID":"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f","Type":"ContainerStarted","Data":"4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228"}
Nov 01 14:55:34.374176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:34.374019    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh" event={"ID":"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f","Type":"ContainerStarted","Data":"4c2af58972ac4579b8c87ee78d5594d035566e7d59617d13aa365e5cd0b72415"}
Nov 01 14:55:34.374238 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:34.374188    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:34.492701 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:34.492648    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh" podStartSLOduration=2.492631867 podStartE2EDuration="2.492631867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:34.489601214 +0000 UTC m=+2274.655086512" watchObservedRunningTime="2026-11-01 14:55:34.492631867 +0000 UTC m=+2274.658117165"
Nov 01 14:55:35.150042 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:55:35.150004    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:55:42.418845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.418809    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"]
Nov 01 14:55:42.419284 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.419061    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb" podUID="459e28fd-ec01-4a69-9491-55cad7526a3f" containerName="maas-api" containerID="cri-o://2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490" gracePeriod=30
Nov 01 14:55:42.722224 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.722201    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:55:42.752552 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.752528    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459e28fd-ec01-4a69-9491-55cad7526a3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459e28fd-ec01-4a69-9491-55cad7526a3f-openshift-service-ca\") pod \"459e28fd-ec01-4a69-9491-55cad7526a3f\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") "
Nov 01 14:55:42.752700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.752560    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-metrics-certs\") pod \"459e28fd-ec01-4a69-9491-55cad7526a3f\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") "
Nov 01 14:55:42.752700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.752622    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls\") pod \"459e28fd-ec01-4a69-9491-55cad7526a3f\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") "
Nov 01 14:55:42.752700 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.752647    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459e28fd-ec01-4a69-9491-55cad7526a3f-kube-api-access-sss6h\" (UniqueName: \"kubernetes.io/projected/459e28fd-ec01-4a69-9491-55cad7526a3f-kube-api-access-sss6h\") pod \"459e28fd-ec01-4a69-9491-55cad7526a3f\" (UID: \"459e28fd-ec01-4a69-9491-55cad7526a3f\") "
Nov 01 14:55:42.752965 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.752938    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459e28fd-ec01-4a69-9491-55cad7526a3f-openshift-service-ca" pod "459e28fd-ec01-4a69-9491-55cad7526a3f" (UID: "459e28fd-ec01-4a69-9491-55cad7526a3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:55:42.754805 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.754781    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-metrics-certs" pod "459e28fd-ec01-4a69-9491-55cad7526a3f" (UID: "459e28fd-ec01-4a69-9491-55cad7526a3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:55:42.754890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.754820    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459e28fd-ec01-4a69-9491-55cad7526a3f-kube-api-access-sss6h" pod "459e28fd-ec01-4a69-9491-55cad7526a3f" (UID: "459e28fd-ec01-4a69-9491-55cad7526a3f"). InnerVolumeSpecName "kube-api-access-sss6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:55:42.754890 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.754870    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls" pod "459e28fd-ec01-4a69-9491-55cad7526a3f" (UID: "459e28fd-ec01-4a69-9491-55cad7526a3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:55:42.854111 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.854073    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459e28fd-ec01-4a69-9491-55cad7526a3f-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:55:42.854111 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.854111    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:55:42.854400 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.854121    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459e28fd-ec01-4a69-9491-55cad7526a3f-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:55:42.854400 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:42.854130    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sss6h\" (UniqueName: \"kubernetes.io/projected/459e28fd-ec01-4a69-9491-55cad7526a3f-kube-api-access-sss6h\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:55:43.433176 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.433139    5423 generic.go:356] "Generic (PLEG): container finished" podID="459e28fd-ec01-4a69-9491-55cad7526a3f" containerID="2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490" exitCode=0
Nov 01 14:55:43.433626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.433187    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"
Nov 01 14:55:43.433626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.433220    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb" event={"ID":"459e28fd-ec01-4a69-9491-55cad7526a3f","Type":"ContainerDied","Data":"2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490"}
Nov 01 14:55:43.433626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.433271    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb" event={"ID":"459e28fd-ec01-4a69-9491-55cad7526a3f","Type":"ContainerDied","Data":"fe952d6f13841d7d575f4d78228157a4ee8b6c490a197ea6b473a1780338bc4c"}
Nov 01 14:55:43.433626 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.433293    5423 scope.go:122] "RemoveContainer" containerID="2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490"
Nov 01 14:55:43.443581 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.443561    5423 scope.go:122] "RemoveContainer" containerID="2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490"
Nov 01 14:55:43.443831 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:55:43.443812    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490\": container with ID starting with 2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490 not found: ID does not exist" containerID="2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490"
Nov 01 14:55:43.443892 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.443843    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490"} err="failed to get container status \"2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490\": rpc error: code = NotFound desc = could not find container \"2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490\": container with ID starting with 2aac660307cf68b2b324f89fb823d1655afa94c16cc7c170eb902fbf2c5a6490 not found: ID does not exist"
Nov 01 14:55:43.601487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.601456    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"]
Nov 01 14:55:43.651113 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.651050    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0bb4b4-7f68746f6-5k4pb"]
Nov 01 14:55:43.961679 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:43.961644    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459e28fd-ec01-4a69-9491-55cad7526a3f" path="/var/lib/kubelet/pods/459e28fd-ec01-4a69-9491-55cad7526a3f/volumes"
Nov 01 14:55:44.375938 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:44.375908    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:55:45.384285 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:45.384259    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:55:46.885667 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:55:46.885626    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:56:02.113976 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.113937    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"]
Nov 01 14:56:02.114547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.114532    5423 cpu_manager.go:414] "containerMap: removing container" podUID="459e28fd-ec01-4a69-9491-55cad7526a3f" containerName="maas-api"
Nov 01 14:56:02.114547 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.114547    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459e28fd-ec01-4a69-9491-55cad7526a3f" containerName="maas-api"
Nov 01 14:56:02.114650 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.114640    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="459e28fd-ec01-4a69-9491-55cad7526a3f" containerName="maas-api"
Nov 01 14:56:02.119409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.119374    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.172746 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.172717    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"]
Nov 01 14:56:02.217208 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.217173    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dsm6\" (UniqueName: \"kubernetes.io/projected/a348208d-d63d-42f3-b681-b76d21d80fac-kube-api-access-7dsm6\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.217432 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.217303    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a348208d-d63d-42f3-b681-b76d21d80fac-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.217501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.217446    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.217501 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.217473    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-metrics-certs\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.221155 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.221132    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7197e2-serving-cert\""
Nov 01 14:56:02.318972 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.318937    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dsm6\" (UniqueName: \"kubernetes.io/projected/a348208d-d63d-42f3-b681-b76d21d80fac-kube-api-access-7dsm6\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.319169 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.319017    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a348208d-d63d-42f3-b681-b76d21d80fac-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.319169 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.319070    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.319169 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.319101    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-metrics-certs\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.319343 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:56:02.319249    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-serving-cert: secret "maas-api-e2e-praxis-b-7197e2-serving-cert" not found
Nov 01 14:56:02.319406 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:56:02.319345    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls podName:a348208d-d63d-42f3-b681-b76d21d80fac nodeName:}" failed. No retries permitted until 2026-11-01 14:56:02.819322718 +0000 UTC m=+2302.984808002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls") pod "maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w" (UID: "a348208d-d63d-42f3-b681-b76d21d80fac") : secret "maas-api-e2e-praxis-b-7197e2-serving-cert" not found
Nov 01 14:56:02.319792 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.319762    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a348208d-d63d-42f3-b681-b76d21d80fac-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.321693 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.321670    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-metrics-certs\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.378905 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.378827    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dsm6\" (UniqueName: \"kubernetes.io/projected/a348208d-d63d-42f3-b681-b76d21d80fac-kube-api-access-7dsm6\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.825543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.825509    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:02.827990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:02.827965    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:03.033179 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:03.033145    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:03.400839 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:56:03.400808    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda348208d_d63d_42f3_b681_b76d21d80fac.slice/crio-e3aab136637b87baae9c07ef047fd4ae56a72785c29fea85b8525c13c3114b6f WatchSource:0}: Error finding container e3aab136637b87baae9c07ef047fd4ae56a72785c29fea85b8525c13c3114b6f: Status 404 returned error can't find the container with id e3aab136637b87baae9c07ef047fd4ae56a72785c29fea85b8525c13c3114b6f
Nov 01 14:56:03.402449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:03.402427    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:56:03.404587 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:03.404566    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"]
Nov 01 14:56:03.588213 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:03.588173    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w" event={"ID":"a348208d-d63d-42f3-b681-b76d21d80fac","Type":"ContainerStarted","Data":"e3aab136637b87baae9c07ef047fd4ae56a72785c29fea85b8525c13c3114b6f"}
Nov 01 14:56:04.596005 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:04.595970    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w" event={"ID":"a348208d-d63d-42f3-b681-b76d21d80fac","Type":"ContainerStarted","Data":"c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9"}
Nov 01 14:56:04.596433 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:04.596112    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:04.715094 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:04.715000    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w" podStartSLOduration=1.75151512 podStartE2EDuration="2.714987279s" totalImagesPullingTime="963.472159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:56:04.712975656 +0000 UTC m=+2304.878460953" watchObservedRunningTime="2026-11-01 14:56:04.714987279 +0000 UTC m=+2304.880472576"
Nov 01 14:56:05.150506 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:56:05.150471    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:56:05.838262 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.838228    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"]
Nov 01 14:56:05.842141 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.842125    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.950014 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.949984    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"]
Nov 01 14:56:05.958761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.958735    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.958911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.958768    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.958911 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.958791    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.959012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.958967    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.959012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.958991    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.959087 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.959040    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.959154 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.959140    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.959226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.959213    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:05.959264 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:05.959234    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c24w\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-kube-api-access-6c24w\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.059810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.059776    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.059810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.059809    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.060033 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.059856    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.060033 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.059906    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.060656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.060449    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.060656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.060522    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c24w\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-kube-api-access-6c24w\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.060656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.060601    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.060656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.060525    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.060656 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.060640    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.061013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.060752    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.061013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.060677    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.061013 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.060768    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.070032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.062600    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.070032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.062849    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.070032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.063549    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.070032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.063957    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.119661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.119608    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.121121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.121097    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c24w\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-kube-api-access-6c24w\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.155812 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.155788    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:06.577959 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.577928    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"]
Nov 01 14:56:06.610688 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:56:06.610645    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa138a7_5cd2_4e74_9ca8_26a40b6d1120.slice/crio-f050be95e236fc3a580bcd80ed593543462ff486b4b668876f209cc90a7d5c92 WatchSource:0}: Error finding container f050be95e236fc3a580bcd80ed593543462ff486b4b668876f209cc90a7d5c92: Status 404 returned error can't find the container with id f050be95e236fc3a580bcd80ed593543462ff486b4b668876f209cc90a7d5c92
Nov 01 14:56:06.613365 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.613333    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:56:06.613464 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.613431    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:56:06.613512 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:06.613463    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:56:07.626130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:07.626090    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" event={"ID":"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120","Type":"ContainerStarted","Data":"5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9"}
Nov 01 14:56:07.626583 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:07.626149    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" event={"ID":"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120","Type":"ContainerStarted","Data":"f050be95e236fc3a580bcd80ed593543462ff486b4b668876f209cc90a7d5c92"}
Nov 01 14:56:07.820012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:07.819920    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podStartSLOduration=2.819895484 podStartE2EDuration="2.819895484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:56:07.80688314 +0000 UTC m=+2307.972368438" watchObservedRunningTime="2026-11-01 14:56:07.819895484 +0000 UTC m=+2307.985380782"
Nov 01 14:56:08.156899 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:08.156867    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:08.158627 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:08.158596    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 14:56:08.158800 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:08.158641    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 14:56:09.156662 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:09.156628    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 14:56:09.157069 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:09.156681    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 14:56:10.156957 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:10.156918    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 14:56:10.157367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:10.156965    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 14:56:10.613520 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:10.613236    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:56:11.156682 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:11.156644    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 14:56:11.156863 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:11.156706    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 14:56:12.156333 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:12.156300    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 14:56:12.156854 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:12.156352    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 14:56:13.156590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:13.156555    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 01 14:56:13.157047 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:13.156613    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 01 14:56:14.160944 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:14.160911    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:14.161442 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:14.161420    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:14.162182 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:14.162161    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 14:56:35.149344 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:56:35.149306    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:56:46.873354 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:46.873320    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:56:56.002299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:56.002220    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt_8819031c-9890-49b6-8677-e37a34d67d25/payload-processing/0.log"
Nov 01 14:56:56.252296 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:56.252269    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wc2s5_0162fda0-7a65-4c47-8aa7-46dd184f8cb5/payload-processing/0.log"
Nov 01 14:56:56.475609 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:56.475583    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wc2s5_0162fda0-7a65-4c47-8aa7-46dd184f8cb5/payload-processing/0.log"
Nov 01 14:56:56.699936 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:56.699907    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w_b8e22105-74fd-42bc-90bc-103772edf87a/payload-processing/0.log"
Nov 01 14:56:56.927624 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:56.927596    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w_b8e22105-74fd-42bc-90bc-103772edf87a/payload-processing/0.log"
Nov 01 14:56:57.634992 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:57.634958    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"]
Nov 01 14:56:57.635410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:56:57.635191    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy" containerID="cri-o://95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7" gracePeriod=30
Nov 01 14:57:02.905296 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:02.905229    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:57:02.984095 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:02.984060    5423 generic.go:356] "Generic (PLEG): container finished" podID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerID="95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7" exitCode=0
Nov 01 14:57:02.984251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:02.984101    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"
Nov 01 14:57:02.984251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:02.984110    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" event={"ID":"89be8ba4-b0f7-43fe-b558-83d407323ed0","Type":"ContainerDied","Data":"95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7"}
Nov 01 14:57:02.984251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:02.984156    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx" event={"ID":"89be8ba4-b0f7-43fe-b558-83d407323ed0","Type":"ContainerDied","Data":"b2ce769b51a634499b9fbdee309512e9d4de232b513f8717c45f584d77253b32"}
Nov 01 14:57:02.984251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:02.984186    5423 scope.go:122] "RemoveContainer" containerID="95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7"
Nov 01 14:57:02.994262 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:02.994244    5423 scope.go:122] "RemoveContainer" containerID="8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5"
Nov 01 14:57:03.003180 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.003163    5423 scope.go:122] "RemoveContainer" containerID="95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7"
Nov 01 14:57:03.003452 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:57:03.003429    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7\": container with ID starting with 95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7 not found: ID does not exist" containerID="95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7"
Nov 01 14:57:03.003509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.003459    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7"} err="failed to get container status \"95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7\": rpc error: code = NotFound desc = could not find container \"95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7\": container with ID starting with 95dbdfef8f757571e2aecc24534528151a652d996e6c68d552a42bfaf9f741d7 not found: ID does not exist"
Nov 01 14:57:03.003509 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.003477    5423 scope.go:122] "RemoveContainer" containerID="8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5"
Nov 01 14:57:03.003705 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:57:03.003691    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5\": container with ID starting with 8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5 not found: ID does not exist" containerID="8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5"
Nov 01 14:57:03.003747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.003708    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5"} err="failed to get container status \"8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5\": rpc error: code = NotFound desc = could not find container \"8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5\": container with ID starting with 8c887f226b97435b3899718cbe9092db2723d40bdf341d13408afd5ae1fa06a5 not found: ID does not exist"
Nov 01 14:57:03.006062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006043    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-token\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-token\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006087    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-envoy\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006113    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-podinfo\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006220    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89be8ba4-b0f7-43fe-b558-83d407323ed0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be8ba4-b0f7-43fe-b558-83d407323ed0-istiod-ca-cert\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006300 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006271    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-credential-socket\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006445 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006424    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-socket\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006575 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006458    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-kube-api-access-2l4d4\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-kube-api-access-2l4d4\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006575 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006545    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-credential-socket" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:03.006698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006587    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89be8ba4-b0f7-43fe-b558-83d407323ed0-istiod-ca-cert" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:57:03.006698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006598    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-data\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006647    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-certs\") pod \"89be8ba4-b0f7-43fe-b558-83d407323ed0\" (UID: \"89be8ba4-b0f7-43fe-b558-83d407323ed0\") "
Nov 01 14:57:03.006698 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006675    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-socket" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:03.006925 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006839    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-data" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:03.007044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.006935    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-certs" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:03.007194 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.007177    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be8ba4-b0f7-43fe-b558-83d407323ed0-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.007256 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.007196    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.007256 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.007213    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.007256 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.007225    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.007256 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.007234    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.008211 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.008176    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-envoy" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:03.008335 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.008319    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-podinfo" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:57:03.008810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.008788    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-token" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:57:03.008810 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.008801    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-kube-api-access-2l4d4" pod "89be8ba4-b0f7-43fe-b558-83d407323ed0" (UID: "89be8ba4-b0f7-43fe-b558-83d407323ed0"). InnerVolumeSpecName "kube-api-access-2l4d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:57:03.108848 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.108817    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l4d4\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-kube-api-access-2l4d4\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.108848 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.108847    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.108848 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.108857    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.109192 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.108867    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be8ba4-b0f7-43fe-b558-83d407323ed0-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:03.409657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.409614    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"]
Nov 01 14:57:03.412942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.412917    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-305290-openshift-default-5cfd7569-qgpfx"]
Nov 01 14:57:03.960937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:03.960905    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" path="/var/lib/kubelet/pods/89be8ba4-b0f7-43fe-b558-83d407323ed0/volumes"
Nov 01 14:57:05.151328 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:57:05.151287    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:57:15.890847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:15.890807    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"]
Nov 01 14:57:15.999363 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:15.999326    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"]
Nov 01 14:57:20.891610 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:20.891560    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7" podUID="b33b5b63-4b7f-414b-add6-fe8e491ba10a" containerName="payload-processing" containerID="cri-o://6932b2a7888b4be16e5c518452c4cb4f02a5ca48d82826afc605b85e16950dff" gracePeriod=25
Nov 01 14:57:20.907572 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:20.907459    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:57:21.006410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.003460    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn" podUID="d0797366-268b-49aa-adcf-ebcd2f3e0f29" containerName="payload-pre-processing" containerID="cri-o://d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99" gracePeriod=25
Nov 01 14:57:21.102543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.102256    5423 generic.go:356] "Generic (PLEG): container finished" podID="b33b5b63-4b7f-414b-add6-fe8e491ba10a" containerID="6932b2a7888b4be16e5c518452c4cb4f02a5ca48d82826afc605b85e16950dff" exitCode=0
Nov 01 14:57:21.102543 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.102478    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7" event={"ID":"b33b5b63-4b7f-414b-add6-fe8e491ba10a","Type":"ContainerDied","Data":"6932b2a7888b4be16e5c518452c4cb4f02a5ca48d82826afc605b85e16950dff"}
Nov 01 14:57:21.283946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.283921    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:57:21.314961 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.314939    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:57:21.477544 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.477472    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"]
Nov 01 14:57:21.478591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.478564    5423 cpu_manager.go:414] "containerMap: removing container" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy"
Nov 01 14:57:21.478787 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.478769    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy"
Nov 01 14:57:21.478894 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.478883    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b33b5b63-4b7f-414b-add6-fe8e491ba10a" containerName="payload-processing"
Nov 01 14:57:21.478990 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.478978    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33b5b63-4b7f-414b-add6-fe8e491ba10a" containerName="payload-processing"
Nov 01 14:57:21.479102 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.479092    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d0797366-268b-49aa-adcf-ebcd2f3e0f29" containerName="payload-pre-processing"
Nov 01 14:57:21.479183 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.479173    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0797366-268b-49aa-adcf-ebcd2f3e0f29" containerName="payload-pre-processing"
Nov 01 14:57:21.479574 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.479555    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0797366-268b-49aa-adcf-ebcd2f3e0f29" containerName="payload-pre-processing"
Nov 01 14:57:21.481503 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.481455    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy"
Nov 01 14:57:21.481584 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.481510    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy"
Nov 01 14:57:21.481584 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.481524    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33b5b63-4b7f-414b-add6-fe8e491ba10a" containerName="payload-processing"
Nov 01 14:57:21.482361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.480689    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33b5b63-4b7f-414b-add6-fe8e491ba10a-plugins-config-volume" pod "b33b5b63-4b7f-414b-add6-fe8e491ba10a" (UID: "b33b5b63-4b7f-414b-add6-fe8e491ba10a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:57:21.482491 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.480259    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33b5b63-4b7f-414b-add6-fe8e491ba10a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33b5b63-4b7f-414b-add6-fe8e491ba10a-plugins-config-volume\") pod \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\" (UID: \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\") "
Nov 01 14:57:21.482669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.482646    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0797366-268b-49aa-adcf-ebcd2f3e0f29-kube-api-access-zhvrd\" (UniqueName: \"kubernetes.io/projected/d0797366-268b-49aa-adcf-ebcd2f3e0f29-kube-api-access-zhvrd\") pod \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\" (UID: \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\") "
Nov 01 14:57:21.482758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.482720    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0797366-268b-49aa-adcf-ebcd2f3e0f29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0797366-268b-49aa-adcf-ebcd2f3e0f29-plugins-config-volume\") pod \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\" (UID: \"d0797366-268b-49aa-adcf-ebcd2f3e0f29\") "
Nov 01 14:57:21.482818 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.482806    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33b5b63-4b7f-414b-add6-fe8e491ba10a-kube-api-access-s42xp\" (UniqueName: \"kubernetes.io/projected/b33b5b63-4b7f-414b-add6-fe8e491ba10a-kube-api-access-s42xp\") pod \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\" (UID: \"b33b5b63-4b7f-414b-add6-fe8e491ba10a\") "
Nov 01 14:57:21.483518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.483205    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0797366-268b-49aa-adcf-ebcd2f3e0f29-plugins-config-volume" pod "d0797366-268b-49aa-adcf-ebcd2f3e0f29" (UID: "d0797366-268b-49aa-adcf-ebcd2f3e0f29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:57:21.483518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.483475    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33b5b63-4b7f-414b-add6-fe8e491ba10a-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:21.483518 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.483497    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0797366-268b-49aa-adcf-ebcd2f3e0f29-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:21.486863 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.486826    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33b5b63-4b7f-414b-add6-fe8e491ba10a-kube-api-access-s42xp" pod "b33b5b63-4b7f-414b-add6-fe8e491ba10a" (UID: "b33b5b63-4b7f-414b-add6-fe8e491ba10a"). InnerVolumeSpecName "kube-api-access-s42xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:57:21.488591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.488556    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0797366-268b-49aa-adcf-ebcd2f3e0f29-kube-api-access-zhvrd" pod "d0797366-268b-49aa-adcf-ebcd2f3e0f29" (UID: "d0797366-268b-49aa-adcf-ebcd2f3e0f29"). InnerVolumeSpecName "kube-api-access-zhvrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:57:21.489711 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.489687    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.564722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.564632    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"]
Nov 01 14:57:21.587421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.585169    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvrd\" (UniqueName: \"kubernetes.io/projected/d0797366-268b-49aa-adcf-ebcd2f3e0f29-kube-api-access-zhvrd\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:21.587421 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.585203    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42xp\" (UniqueName: \"kubernetes.io/projected/b33b5b63-4b7f-414b-add6-fe8e491ba10a-kube-api-access-s42xp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:21.686525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.686487    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.686708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.686556    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66320d1e-93ec-46ac-8bfd-623f4605a96a-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.686708 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.686685    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.686799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.686755    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.686850 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.686796    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.686974 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.686934    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.687057 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.687025    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.687133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.687108    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz6wl\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-kube-api-access-cz6wl\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.687190 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.687150    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.788568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788473    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.788568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788549    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz6wl\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-kube-api-access-cz6wl\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.788807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788578    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.788807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788635    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.788807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788705    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66320d1e-93ec-46ac-8bfd-623f4605a96a-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.788807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788736    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.788807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788780    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.788807 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788804    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.789121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.788866    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.789121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.789085    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.789707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.789678    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.789909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.789885    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66320d1e-93ec-46ac-8bfd-623f4605a96a-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.790029 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.789857    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.790029 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.789743    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.791600 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.791567    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.791845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.791823    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.847268 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.847230    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:21.847578 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:21.847558    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz6wl\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-kube-api-access-cz6wl\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:22.112333 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.112302    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"
Nov 01 14:57:22.113114 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.112340    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7" event={"ID":"b33b5b63-4b7f-414b-add6-fe8e491ba10a","Type":"ContainerDied","Data":"571847c3ecc8647f9f6c7fbe91b87b7c91947d6f39307743ed8dc27989c904e8"}
Nov 01 14:57:22.113221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.113182    5423 scope.go:122] "RemoveContainer" containerID="6932b2a7888b4be16e5c518452c4cb4f02a5ca48d82826afc605b85e16950dff"
Nov 01 14:57:22.116760 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.116165    5423 generic.go:356] "Generic (PLEG): container finished" podID="d0797366-268b-49aa-adcf-ebcd2f3e0f29" containerID="d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99" exitCode=0
Nov 01 14:57:22.116760 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.116226    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn" event={"ID":"d0797366-268b-49aa-adcf-ebcd2f3e0f29","Type":"ContainerDied","Data":"d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99"}
Nov 01 14:57:22.116760 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.116283    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn" event={"ID":"d0797366-268b-49aa-adcf-ebcd2f3e0f29","Type":"ContainerDied","Data":"39fed56cb0c83988897e17e889a79b6ef5cccaf7a80bec8821a396dee035f95c"}
Nov 01 14:57:22.117309 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.117133    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"
Nov 01 14:57:22.117997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.117979    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:22.172272 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.172243    5423 scope.go:122] "RemoveContainer" containerID="d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99"
Nov 01 14:57:22.315408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.308978    5423 scope.go:122] "RemoveContainer" containerID="d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99"
Nov 01 14:57:22.315408 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:57:22.314836    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99\": container with ID starting with d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99 not found: ID does not exist" containerID="d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99"
Nov 01 14:57:22.315408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.314880    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99"} err="failed to get container status \"d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99\": rpc error: code = NotFound desc = could not find container \"d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99\": container with ID starting with d4ce8a368ebfda116aa7966bb50b7bb454deb3489562a874f60dc262c0a4ff99 not found: ID does not exist"
Nov 01 14:57:22.352415 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.351576    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"]
Nov 01 14:57:22.353750 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.353723    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a90080-669dfcc667-hbrd7"]
Nov 01 14:57:22.478157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.478119    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"]
Nov 01 14:57:22.483431 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.483408    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a90080-585c75769f-zrgqn"]
Nov 01 14:57:22.787746 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:57:22.787712    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66320d1e_93ec_46ac_8bfd_623f4605a96a.slice/crio-ccfffa49d5ac0bb3bdcfb48d2a097ed4711a8060b8438c6588f9bfc4ec631e17 WatchSource:0}: Error finding container ccfffa49d5ac0bb3bdcfb48d2a097ed4711a8060b8438c6588f9bfc4ec631e17: Status 404 returned error can't find the container with id ccfffa49d5ac0bb3bdcfb48d2a097ed4711a8060b8438c6588f9bfc4ec631e17
Nov 01 14:57:22.790520 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.790484    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:57:22.790632 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.790567    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:57:22.790632 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.790613    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:57:22.791024 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:22.790997    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"]
Nov 01 14:57:23.129059 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:23.129020    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" event={"ID":"66320d1e-93ec-46ac-8bfd-623f4605a96a","Type":"ContainerStarted","Data":"8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d"}
Nov 01 14:57:23.129592 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:23.129082    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" event={"ID":"66320d1e-93ec-46ac-8bfd-623f4605a96a","Type":"ContainerStarted","Data":"ccfffa49d5ac0bb3bdcfb48d2a097ed4711a8060b8438c6588f9bfc4ec631e17"}
Nov 01 14:57:23.960714 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:23.960681    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33b5b63-4b7f-414b-add6-fe8e491ba10a" path="/var/lib/kubelet/pods/b33b5b63-4b7f-414b-add6-fe8e491ba10a/volumes"
Nov 01 14:57:23.961133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:23.961118    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0797366-268b-49aa-adcf-ebcd2f3e0f29" path="/var/lib/kubelet/pods/d0797366-268b-49aa-adcf-ebcd2f3e0f29/volumes"
Nov 01 14:57:24.119409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:24.118734    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:24.120181 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:24.120147    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 14:57:24.120304 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:24.120208    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 14:57:25.119799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:25.119703    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 14:57:25.119799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:25.119765    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 14:57:26.120134 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:26.119693    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 14:57:26.120134 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:26.119754    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 14:57:27.119541 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:27.119497    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 14:57:27.119758 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:27.119584    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 14:57:28.119553 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:28.119512    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 14:57:28.120023 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:28.119579    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 14:57:29.118804 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:29.118765    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 14:57:29.118993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:29.118837    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 14:57:30.194120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:30.194086    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:30.195135 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:30.195107    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:30.196026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:30.196003    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 14:57:30.416940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:30.416887    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podStartSLOduration=9.416874871 podStartE2EDuration="9.416874871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:23.252432581 +0000 UTC m=+2383.417917879" watchObservedRunningTime="2026-11-01 14:57:30.416874871 +0000 UTC m=+2390.582360169"
Nov 01 14:57:35.150495 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:57:35.150455    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:57:42.503075 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:42.503043    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:57:42.536740 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:42.536717    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 14:57:44.023766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:44.023727    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"]
Nov 01 14:57:44.024219 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:44.023997    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" containerID="cri-o://013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad" gracePeriod=30
Nov 01 14:57:48.764062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:48.764023    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l8f9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:57:48.764586 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:48.764110    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:57:49.296035 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.295970    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:57:49.315696 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.315663    5423 generic.go:356] "Generic (PLEG): container finished" podID="19eaa467-1361-4dac-acde-767aad1eadd0" containerID="013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad" exitCode=0
Nov 01 14:57:49.315830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.315716    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"
Nov 01 14:57:49.315830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.315738    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" event={"ID":"19eaa467-1361-4dac-acde-767aad1eadd0","Type":"ContainerDied","Data":"013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad"}
Nov 01 14:57:49.315830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.315793    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k" event={"ID":"19eaa467-1361-4dac-acde-767aad1eadd0","Type":"ContainerDied","Data":"632bb254108c159814af419d6076f6bafedd27aaaeb8b84ef29ad5f777977111"}
Nov 01 14:57:49.315830 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.315814    5423 scope.go:122] "RemoveContainer" containerID="013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad"
Nov 01 14:57:49.325063 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.325045    5423 scope.go:122] "RemoveContainer" containerID="013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad"
Nov 01 14:57:49.325309 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:57:49.325293    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad\": container with ID starting with 013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad not found: ID does not exist" containerID="013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad"
Nov 01 14:57:49.325360 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.325320    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad"} err="failed to get container status \"013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad\": rpc error: code = NotFound desc = could not find container \"013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad\": container with ID starting with 013c4849646173d68ffe5781fea06d47c546ff3c23f3edd57dbb2b8be7e7d3ad not found: ID does not exist"
Nov 01 14:57:49.329761 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.329738    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-socket\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.329852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.329774    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-istio-token\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-istio-token\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.329852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.329793    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-envoy\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.329852 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.329830    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-certs\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.329953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.329914    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-data\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.329953 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.329949    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19eaa467-1361-4dac-acde-767aad1eadd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19eaa467-1361-4dac-acde-767aad1eadd0-istio-podinfo\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.330033 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.329988    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-socket" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:49.330033 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330002    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-kube-api-access-v6dtj\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-kube-api-access-v6dtj\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.330143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330071    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19eaa467-1361-4dac-acde-767aad1eadd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19eaa467-1361-4dac-acde-767aad1eadd0-istiod-ca-cert\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.330143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330133    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-credential-socket\") pod \"19eaa467-1361-4dac-acde-767aad1eadd0\" (UID: \"19eaa467-1361-4dac-acde-767aad1eadd0\") "
Nov 01 14:57:49.330247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330193    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-certs" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:49.330247 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330217    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-data" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:49.330607 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330531    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19eaa467-1361-4dac-acde-767aad1eadd0-istiod-ca-cert" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:57:49.330607 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330576    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-credential-socket" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:49.330766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330746    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.330803 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330775    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19eaa467-1361-4dac-acde-767aad1eadd0-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.330803 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330792    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.330868 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330808    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.330868 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.330823    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.332046 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.332020    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-envoy" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:49.332046 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.332041    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-istio-token" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:57:49.332530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.332511    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19eaa467-1361-4dac-acde-767aad1eadd0-istio-podinfo" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:57:49.332613 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.332540    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-kube-api-access-v6dtj" pod "19eaa467-1361-4dac-acde-767aad1eadd0" (UID: "19eaa467-1361-4dac-acde-767aad1eadd0"). InnerVolumeSpecName "kube-api-access-v6dtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:57:49.431956 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.431925    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.431956 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.431954    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19eaa467-1361-4dac-acde-767aad1eadd0-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.432154 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.431964    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19eaa467-1361-4dac-acde-767aad1eadd0-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.432154 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.431974    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6dtj\" (UniqueName: \"kubernetes.io/projected/19eaa467-1361-4dac-acde-767aad1eadd0-kube-api-access-v6dtj\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:49.873651 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.873619    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"]
Nov 01 14:57:49.875722 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.875675    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l8f9k"]
Nov 01 14:57:49.961242 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.961213    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" path="/var/lib/kubelet/pods/19eaa467-1361-4dac-acde-767aad1eadd0/volumes"
Nov 01 14:57:49.972607 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.972581    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"]
Nov 01 14:57:49.972760 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:49.972740    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29" podUID="12926b0f-5040-45fa-b1cd-b99c4e784c2b" containerName="maas-api" containerID="cri-o://d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e" gracePeriod=30
Nov 01 14:57:50.311012 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.310988    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:57:50.373450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.371105    5423 generic.go:356] "Generic (PLEG): container finished" podID="12926b0f-5040-45fa-b1cd-b99c4e784c2b" containerID="d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e" exitCode=0
Nov 01 14:57:50.373450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.371263    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"
Nov 01 14:57:50.373450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.371448    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29" event={"ID":"12926b0f-5040-45fa-b1cd-b99c4e784c2b","Type":"ContainerDied","Data":"d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e"}
Nov 01 14:57:50.373450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.371485    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29" event={"ID":"12926b0f-5040-45fa-b1cd-b99c4e784c2b","Type":"ContainerDied","Data":"48455a4e9dfdc9da604a3e941cec4fa53a433a5ee36c7d5a505c93d77a2ed099"}
Nov 01 14:57:50.373450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.371508    5423 scope.go:122] "RemoveContainer" containerID="d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e"
Nov 01 14:57:50.388146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.388123    5423 scope.go:122] "RemoveContainer" containerID="d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e"
Nov 01 14:57:50.389122 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:57:50.388721    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e\": container with ID starting with d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e not found: ID does not exist" containerID="d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e"
Nov 01 14:57:50.389122 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.388765    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e"} err="failed to get container status \"d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e\": rpc error: code = NotFound desc = could not find container \"d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e\": container with ID starting with d5aa90512d8333d9c10985206a9c80e819485cea99ea0cdb1f2f4c14d1bbff1e not found: ID does not exist"
Nov 01 14:57:50.443490 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.443411    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls\") pod \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") "
Nov 01 14:57:50.443664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.443598    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-metrics-certs\") pod \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") "
Nov 01 14:57:50.443664 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.443646    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12926b0f-5040-45fa-b1cd-b99c4e784c2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12926b0f-5040-45fa-b1cd-b99c4e784c2b-openshift-service-ca\") pod \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") "
Nov 01 14:57:50.443788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.443680    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12926b0f-5040-45fa-b1cd-b99c4e784c2b-kube-api-access-w4hc8\" (UniqueName: \"kubernetes.io/projected/12926b0f-5040-45fa-b1cd-b99c4e784c2b-kube-api-access-w4hc8\") pod \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\" (UID: \"12926b0f-5040-45fa-b1cd-b99c4e784c2b\") "
Nov 01 14:57:50.443991 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.443966    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12926b0f-5040-45fa-b1cd-b99c4e784c2b-openshift-service-ca" pod "12926b0f-5040-45fa-b1cd-b99c4e784c2b" (UID: "12926b0f-5040-45fa-b1cd-b99c4e784c2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:57:50.444223 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.444203    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12926b0f-5040-45fa-b1cd-b99c4e784c2b-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:50.446212 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.446178    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-metrics-certs" pod "12926b0f-5040-45fa-b1cd-b99c4e784c2b" (UID: "12926b0f-5040-45fa-b1cd-b99c4e784c2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:57:50.446628 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.446601    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12926b0f-5040-45fa-b1cd-b99c4e784c2b-kube-api-access-w4hc8" pod "12926b0f-5040-45fa-b1cd-b99c4e784c2b" (UID: "12926b0f-5040-45fa-b1cd-b99c4e784c2b"). InnerVolumeSpecName "kube-api-access-w4hc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:57:50.446773 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.446709    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls" pod "12926b0f-5040-45fa-b1cd-b99c4e784c2b" (UID: "12926b0f-5040-45fa-b1cd-b99c4e784c2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:57:50.545871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.545837    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:50.545871 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.545868    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hc8\" (UniqueName: \"kubernetes.io/projected/12926b0f-5040-45fa-b1cd-b99c4e784c2b-kube-api-access-w4hc8\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:50.546061 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.545881    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12926b0f-5040-45fa-b1cd-b99c4e784c2b-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:50.794293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.794216    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"]
Nov 01 14:57:50.798127 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:50.798100    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a90080-6db754c75b-htd29"]
Nov 01 14:57:51.965900 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:51.965861    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12926b0f-5040-45fa-b1cd-b99c4e784c2b" path="/var/lib/kubelet/pods/12926b0f-5040-45fa-b1cd-b99c4e784c2b/volumes"
Nov 01 14:57:55.378262 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.378228    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"]
Nov 01 14:57:55.378993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.378972    5423 cpu_manager.go:414] "containerMap: removing container" podUID="12926b0f-5040-45fa-b1cd-b99c4e784c2b" containerName="maas-api"
Nov 01 14:57:55.378993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.378995    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12926b0f-5040-45fa-b1cd-b99c4e784c2b" containerName="maas-api"
Nov 01 14:57:55.379117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.379026    5423 cpu_manager.go:414] "containerMap: removing container" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy"
Nov 01 14:57:55.379117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.379031    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89be8ba4-b0f7-43fe-b558-83d407323ed0" containerName="istio-proxy"
Nov 01 14:57:55.379117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.379056    5423 cpu_manager.go:414] "containerMap: removing container" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy"
Nov 01 14:57:55.379117 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.379061    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy"
Nov 01 14:57:55.379255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.379153    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="19eaa467-1361-4dac-acde-767aad1eadd0" containerName="istio-proxy"
Nov 01 14:57:55.379255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.379163    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="12926b0f-5040-45fa-b1cd-b99c4e784c2b" containerName="maas-api"
Nov 01 14:57:55.383910 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.383888    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.399964 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.399941    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.400062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.399993    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9v7c\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-kube-api-access-r9v7c\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.400124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.400107    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.400183 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.400163    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.400274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.400254    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.400333 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.400312    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.400414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.400398    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.400480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.400425    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.400480 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.400453    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.491782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.491750    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"]
Nov 01 14:57:55.502152 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502117    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502181    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502230    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502303 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502290    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502502 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502332    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502502 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502367    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502502 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502437    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502527    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502557    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502590    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9v7c\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-kube-api-access-r9v7c\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502674 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502632    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502696    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.502895 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.502755    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.503206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.503186    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.505419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.505377    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.505558 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.505544    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.560222 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.560195    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.560321 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.560304    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9v7c\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-kube-api-access-r9v7c\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:55.697608 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:55.697536    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:56.129221 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:56.129188    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"]
Nov 01 14:57:56.159655 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:57:56.159628    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e422494_0e14_4a7d_83e6_c29e2030d6bd.slice/crio-8c3ab47a0dcd4280258380a4589da7a34985bcca84465b967e975aa3c5a95761 WatchSource:0}: Error finding container 8c3ab47a0dcd4280258380a4589da7a34985bcca84465b967e975aa3c5a95761: Status 404 returned error can't find the container with id 8c3ab47a0dcd4280258380a4589da7a34985bcca84465b967e975aa3c5a95761
Nov 01 14:57:56.162206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:56.162172    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:57:56.162300 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:56.162256    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:57:56.162300 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:56.162284    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:57:56.452508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:56.444869    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" event={"ID":"8e422494-0e14-4a7d-83e6-c29e2030d6bd","Type":"ContainerStarted","Data":"544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1"}
Nov 01 14:57:56.452508 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:56.444941    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" event={"ID":"8e422494-0e14-4a7d-83e6-c29e2030d6bd","Type":"ContainerStarted","Data":"8c3ab47a0dcd4280258380a4589da7a34985bcca84465b967e975aa3c5a95761"}
Nov 01 14:57:56.638425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:56.637056    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" podStartSLOduration=1.637019768 podStartE2EDuration="1.637019768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:56.634646304 +0000 UTC m=+2416.800131613" watchObservedRunningTime="2026-11-01 14:57:56.637019768 +0000 UTC m=+2416.802505102"
Nov 01 14:57:56.697787 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:56.697751    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:57:57.712411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:57.704529    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 01 14:57:57.712411 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:57.704603    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 01 14:57:58.710410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:58.702779    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 01 14:57:58.710410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:58.702856    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 01 14:57:59.714410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:59.711733    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 01 14:57:59.714410 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:57:59.711796    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 01 14:58:00.714802 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:00.714760    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 01 14:58:00.716647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:00.716614    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 01 14:58:01.719829 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:01.719786    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 01 14:58:01.720253 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:01.719844    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 01 14:58:02.698823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:02.698781    5423 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 01 14:58:02.699019 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:02.698853    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 01 14:58:03.702921 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:03.702888    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:58:03.703429 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:03.703282    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:58:03.703983 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:03.703962    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 14:58:05.149696 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:58:05.149658    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:58:09.165449 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:09.165404    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:58:09.526340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:09.525167    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"]
Nov 01 14:58:09.526340 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:09.525498    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml" podUID="b6dcd1d6-f381-4556-ba66-18197a1648a6" containerName="maas-api" containerID="cri-o://f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163" gracePeriod=30
Nov 01 14:58:09.929322 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:09.929289    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:58:09.946987 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:09.946953    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"]
Nov 01 14:58:09.947226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:09.947203    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt" podUID="8819031c-9890-49b6-8677-e37a34d67d25" containerName="payload-processing" containerID="cri-o://bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190" gracePeriod=30
Nov 01 14:58:10.001474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.001439    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6dcd1d6-f381-4556-ba66-18197a1648a6-kube-api-access-p9t2l\" (UniqueName: \"kubernetes.io/projected/b6dcd1d6-f381-4556-ba66-18197a1648a6-kube-api-access-p9t2l\") pod \"b6dcd1d6-f381-4556-ba66-18197a1648a6\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") "
Nov 01 14:58:10.001629 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.001486    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-metrics-certs\") pod \"b6dcd1d6-f381-4556-ba66-18197a1648a6\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") "
Nov 01 14:58:10.001725 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.001706    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6dcd1d6-f381-4556-ba66-18197a1648a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dcd1d6-f381-4556-ba66-18197a1648a6-openshift-service-ca\") pod \"b6dcd1d6-f381-4556-ba66-18197a1648a6\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") "
Nov 01 14:58:10.001797 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.001759    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls\") pod \"b6dcd1d6-f381-4556-ba66-18197a1648a6\" (UID: \"b6dcd1d6-f381-4556-ba66-18197a1648a6\") "
Nov 01 14:58:10.002132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.002101    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6dcd1d6-f381-4556-ba66-18197a1648a6-openshift-service-ca" pod "b6dcd1d6-f381-4556-ba66-18197a1648a6" (UID: "b6dcd1d6-f381-4556-ba66-18197a1648a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:58:10.002559 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.002533    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6dcd1d6-f381-4556-ba66-18197a1648a6-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:10.004245 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.004210    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-metrics-certs" pod "b6dcd1d6-f381-4556-ba66-18197a1648a6" (UID: "b6dcd1d6-f381-4556-ba66-18197a1648a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:58:10.004459 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.004440    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls" pod "b6dcd1d6-f381-4556-ba66-18197a1648a6" (UID: "b6dcd1d6-f381-4556-ba66-18197a1648a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:58:10.004530 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.004456    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6dcd1d6-f381-4556-ba66-18197a1648a6-kube-api-access-p9t2l" pod "b6dcd1d6-f381-4556-ba66-18197a1648a6" (UID: "b6dcd1d6-f381-4556-ba66-18197a1648a6"). InnerVolumeSpecName "kube-api-access-p9t2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:10.025369 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.025352    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"]
Nov 01 14:58:10.025568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.025548    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt" podUID="b7988724-6fe6-41d0-8f01-626458fe98f0" containerName="payload-pre-processing" containerID="cri-o://91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d" gracePeriod=30
Nov 01 14:58:10.106483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.105848    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9t2l\" (UniqueName: \"kubernetes.io/projected/b6dcd1d6-f381-4556-ba66-18197a1648a6-kube-api-access-p9t2l\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:10.106483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.105890    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:10.106483 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.105906    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6dcd1d6-f381-4556-ba66-18197a1648a6-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:10.360896 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.360747    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:58:10.409431 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.404846    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:58:10.415909 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.415880    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8819031c-9890-49b6-8677-e37a34d67d25-kube-api-access-fmddr\" (UniqueName: \"kubernetes.io/projected/8819031c-9890-49b6-8677-e37a34d67d25-kube-api-access-fmddr\") pod \"8819031c-9890-49b6-8677-e37a34d67d25\" (UID: \"8819031c-9890-49b6-8677-e37a34d67d25\") "
Nov 01 14:58:10.416614 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.416593    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8819031c-9890-49b6-8677-e37a34d67d25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8819031c-9890-49b6-8677-e37a34d67d25-plugins-config-volume\") pod \"8819031c-9890-49b6-8677-e37a34d67d25\" (UID: \"8819031c-9890-49b6-8677-e37a34d67d25\") "
Nov 01 14:58:10.417979 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.417956    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8819031c-9890-49b6-8677-e37a34d67d25-plugins-config-volume" pod "8819031c-9890-49b6-8677-e37a34d67d25" (UID: "8819031c-9890-49b6-8677-e37a34d67d25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:58:10.423332 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.423308    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8819031c-9890-49b6-8677-e37a34d67d25-kube-api-access-fmddr" pod "8819031c-9890-49b6-8677-e37a34d67d25" (UID: "8819031c-9890-49b6-8677-e37a34d67d25"). InnerVolumeSpecName "kube-api-access-fmddr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:10.622457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.554552    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7988724-6fe6-41d0-8f01-626458fe98f0-kube-api-access-zsksm\" (UniqueName: \"kubernetes.io/projected/b7988724-6fe6-41d0-8f01-626458fe98f0-kube-api-access-zsksm\") pod \"b7988724-6fe6-41d0-8f01-626458fe98f0\" (UID: \"b7988724-6fe6-41d0-8f01-626458fe98f0\") "
Nov 01 14:58:10.622457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.566674    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7988724-6fe6-41d0-8f01-626458fe98f0-plugins-config-volume" pod "b7988724-6fe6-41d0-8f01-626458fe98f0" (UID: "b7988724-6fe6-41d0-8f01-626458fe98f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:58:10.622457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.566784    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7988724-6fe6-41d0-8f01-626458fe98f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7988724-6fe6-41d0-8f01-626458fe98f0-plugins-config-volume\") pod \"b7988724-6fe6-41d0-8f01-626458fe98f0\" (UID: \"b7988724-6fe6-41d0-8f01-626458fe98f0\") "
Nov 01 14:58:10.622457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.567302    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmddr\" (UniqueName: \"kubernetes.io/projected/8819031c-9890-49b6-8677-e37a34d67d25-kube-api-access-fmddr\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:10.622457 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.567324    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8819031c-9890-49b6-8677-e37a34d67d25-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.643655    5423 generic.go:356] "Generic (PLEG): container finished" podID="b6dcd1d6-f381-4556-ba66-18197a1648a6" containerID="f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163" exitCode=0
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.643762    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml" event={"ID":"b6dcd1d6-f381-4556-ba66-18197a1648a6","Type":"ContainerDied","Data":"f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163"}
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.643818    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml" event={"ID":"b6dcd1d6-f381-4556-ba66-18197a1648a6","Type":"ContainerDied","Data":"e94af8632d53d6cd79262a43a603a40d158d1f4e14a6975e4cd7045fe526c086"}
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.643848    5423 scope.go:122] "RemoveContainer" containerID="f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163"
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.644011    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.660211    5423 generic.go:356] "Generic (PLEG): container finished" podID="8819031c-9890-49b6-8677-e37a34d67d25" containerID="bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190" exitCode=0
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.660320    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt" event={"ID":"8819031c-9890-49b6-8677-e37a34d67d25","Type":"ContainerDied","Data":"bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190"}
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.660371    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt" event={"ID":"8819031c-9890-49b6-8677-e37a34d67d25","Type":"ContainerDied","Data":"529ec068141cc733b4d59ec1142f29136dabf350d5a04180d390ce48bc541f04"}
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.660512    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.662115    5423 generic.go:356] "Generic (PLEG): container finished" podID="b7988724-6fe6-41d0-8f01-626458fe98f0" containerID="91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d" exitCode=0
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.662172    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt" event={"ID":"b7988724-6fe6-41d0-8f01-626458fe98f0","Type":"ContainerDied","Data":"91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d"}
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.662211    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt" event={"ID":"b7988724-6fe6-41d0-8f01-626458fe98f0","Type":"ContainerDied","Data":"df3fc0eed861c743892cf596d38237d94607fedd1f9625618488bccf47550b7f"}
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.662278    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.668490    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7988724-6fe6-41d0-8f01-626458fe98f0-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:10.691453 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.668639    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7988724-6fe6-41d0-8f01-626458fe98f0-kube-api-access-zsksm" pod "b7988724-6fe6-41d0-8f01-626458fe98f0" (UID: "b7988724-6fe6-41d0-8f01-626458fe98f0"). InnerVolumeSpecName "kube-api-access-zsksm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:10.775477 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:10.770044    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsksm\" (UniqueName: \"kubernetes.io/projected/b7988724-6fe6-41d0-8f01-626458fe98f0-kube-api-access-zsksm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:11.078048 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.078008    5423 scope.go:122] "RemoveContainer" containerID="f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163"
Nov 01 14:58:11.079234 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:58:11.079210    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163\": container with ID starting with f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163 not found: ID does not exist" containerID="f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163"
Nov 01 14:58:11.079824 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.079788    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163"} err="failed to get container status \"f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163\": rpc error: code = NotFound desc = could not find container \"f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163\": container with ID starting with f78a0a50bf247632b0fc7a25f848650d4e19770dc1ef42f5113be2b3b566d163 not found: ID does not exist"
Nov 01 14:58:11.079978 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.079963    5423 scope.go:122] "RemoveContainer" containerID="bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190"
Nov 01 14:58:11.089066 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.089045    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"]
Nov 01 14:58:11.093000 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.092979    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-5c8f945dd6-776vt"]
Nov 01 14:58:11.120407 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.120361    5423 scope.go:122] "RemoveContainer" containerID="bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190"
Nov 01 14:58:11.121097 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:58:11.121072    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190\": container with ID starting with bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190 not found: ID does not exist" containerID="bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190"
Nov 01 14:58:11.121321 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.121271    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190"} err="failed to get container status \"bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190\": rpc error: code = NotFound desc = could not find container \"bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190\": container with ID starting with bd3861e315a084d841c6b71cdae144b4f4bfbfa8a1431cc0866f304180266190 not found: ID does not exist"
Nov 01 14:58:11.121465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.121450    5423 scope.go:122] "RemoveContainer" containerID="91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d"
Nov 01 14:58:11.142073 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.142052    5423 scope.go:122] "RemoveContainer" containerID="91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d"
Nov 01 14:58:11.143039 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:58:11.142990    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d\": container with ID starting with 91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d not found: ID does not exist" containerID="91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d"
Nov 01 14:58:11.143239 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.143211    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d"} err="failed to get container status \"91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d\": rpc error: code = NotFound desc = could not find container \"91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d\": container with ID starting with 91793520e52572e4466d4bc10d15f8ddcc0aab1cafa7e227e5208cb93178e59d not found: ID does not exist"
Nov 01 14:58:11.196229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.196149    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"]
Nov 01 14:58:11.199770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.199741    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2048c1-79f5b5ff79-nlprt"]
Nov 01 14:58:11.307801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.307760    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"]
Nov 01 14:58:11.314488 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.314454    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2048c1-5c8dcd5d9d-clcml"]
Nov 01 14:58:11.985146 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.985109    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8819031c-9890-49b6-8677-e37a34d67d25" path="/var/lib/kubelet/pods/8819031c-9890-49b6-8677-e37a34d67d25/volumes"
Nov 01 14:58:11.985828 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.985798    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6dcd1d6-f381-4556-ba66-18197a1648a6" path="/var/lib/kubelet/pods/b6dcd1d6-f381-4556-ba66-18197a1648a6/volumes"
Nov 01 14:58:11.986366 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:11.986345    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7988724-6fe6-41d0-8f01-626458fe98f0" path="/var/lib/kubelet/pods/b7988724-6fe6-41d0-8f01-626458fe98f0/volumes"
Nov 01 14:58:28.076817 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:28.076733    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"]
Nov 01 14:58:33.077899 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:33.077836    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz" podUID="cbed19b3-8946-4aa8-81b0-8d5874180d70" containerName="payload-processing" containerID="cri-o://ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a" gracePeriod=25
Nov 01 14:58:35.149576 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:58:35.149541    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:58:36.320462 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:36.320426    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz" podUID="cbed19b3-8946-4aa8-81b0-8d5874180d70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:58:45.564940 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:45.564905    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"]
Nov 01 14:58:45.565413 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:45.565092    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w" podUID="a348208d-d63d-42f3-b681-b76d21d80fac" containerName="maas-api" containerID="cri-o://c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9" gracePeriod=30
Nov 01 14:58:45.602733 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:45.602704    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w" podUID="a348208d-d63d-42f3-b681-b76d21d80fac" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.152:8443/health\": dial tcp 10.133.0.152:8443: connect: connection refused"
Nov 01 14:58:45.919747 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:45.910402    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:58:45.998563 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:45.998521    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a348208d-d63d-42f3-b681-b76d21d80fac-kube-api-access-7dsm6\" (UniqueName: \"kubernetes.io/projected/a348208d-d63d-42f3-b681-b76d21d80fac-kube-api-access-7dsm6\") pod \"a348208d-d63d-42f3-b681-b76d21d80fac\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") "
Nov 01 14:58:45.998770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:45.998647    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls\") pod \"a348208d-d63d-42f3-b681-b76d21d80fac\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") "
Nov 01 14:58:45.998770 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:45.998735    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a348208d-d63d-42f3-b681-b76d21d80fac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a348208d-d63d-42f3-b681-b76d21d80fac-openshift-service-ca\") pod \"a348208d-d63d-42f3-b681-b76d21d80fac\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") "
Nov 01 14:58:45.998893 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:45.998860    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-metrics-certs\") pod \"a348208d-d63d-42f3-b681-b76d21d80fac\" (UID: \"a348208d-d63d-42f3-b681-b76d21d80fac\") "
Nov 01 14:58:46.005226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.005145    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls" pod "a348208d-d63d-42f3-b681-b76d21d80fac" (UID: "a348208d-d63d-42f3-b681-b76d21d80fac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:58:46.005474 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.005171    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a348208d-d63d-42f3-b681-b76d21d80fac-openshift-service-ca" pod "a348208d-d63d-42f3-b681-b76d21d80fac" (UID: "a348208d-d63d-42f3-b681-b76d21d80fac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:58:46.013305 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.013250    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a348208d-d63d-42f3-b681-b76d21d80fac-kube-api-access-7dsm6" pod "a348208d-d63d-42f3-b681-b76d21d80fac" (UID: "a348208d-d63d-42f3-b681-b76d21d80fac"). InnerVolumeSpecName "kube-api-access-7dsm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:46.014767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.014740    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-metrics-certs" pod "a348208d-d63d-42f3-b681-b76d21d80fac" (UID: "a348208d-d63d-42f3-b681-b76d21d80fac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:58:46.040381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.037053    5423 generic.go:356] "Generic (PLEG): container finished" podID="a348208d-d63d-42f3-b681-b76d21d80fac" containerID="c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9" exitCode=0
Nov 01 14:58:46.040381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.037127    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"
Nov 01 14:58:46.040381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.037209    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w" event={"ID":"a348208d-d63d-42f3-b681-b76d21d80fac","Type":"ContainerDied","Data":"c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9"}
Nov 01 14:58:46.040381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.037255    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w" event={"ID":"a348208d-d63d-42f3-b681-b76d21d80fac","Type":"ContainerDied","Data":"e3aab136637b87baae9c07ef047fd4ae56a72785c29fea85b8525c13c3114b6f"}
Nov 01 14:58:46.040381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.037283    5423 scope.go:122] "RemoveContainer" containerID="c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9"
Nov 01 14:58:46.060542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.060523    5423 scope.go:122] "RemoveContainer" containerID="c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9"
Nov 01 14:58:46.060886 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:58:46.060862    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9\": container with ID starting with c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9 not found: ID does not exist" containerID="c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9"
Nov 01 14:58:46.060986 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.060893    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9"} err="failed to get container status \"c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9\": rpc error: code = NotFound desc = could not find container \"c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9\": container with ID starting with c68f230e6632093ac95d2876415e6e867a4802a344a86eb719c1966ba45762f9 not found: ID does not exist"
Nov 01 14:58:46.100897 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.100862    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dsm6\" (UniqueName: \"kubernetes.io/projected/a348208d-d63d-42f3-b681-b76d21d80fac-kube-api-access-7dsm6\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:46.100897 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.100898    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:46.101064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.100917    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a348208d-d63d-42f3-b681-b76d21d80fac-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:46.101064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.100933    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a348208d-d63d-42f3-b681-b76d21d80fac-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:46.209899 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.209817    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"]
Nov 01 14:58:46.214199 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.214170    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7197e2-5cc75fc578-mvx5w"]
Nov 01 14:58:46.320836 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.320804    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz" podUID="cbed19b3-8946-4aa8-81b0-8d5874180d70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 14:58:46.626359 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.626324    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"]
Nov 01 14:58:46.626766 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.626534    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv" podUID="197dcbfb-9678-4806-bee1-4d57480b02f6" containerName="maas-api" containerID="cri-o://49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791" gracePeriod=30
Nov 01 14:58:46.934265 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:46.934244    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:58:47.013834 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.013795    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-maas-api-tls\") pod \"197dcbfb-9678-4806-bee1-4d57480b02f6\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") "
Nov 01 14:58:47.014555 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.014529    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-metrics-certs\") pod \"197dcbfb-9678-4806-bee1-4d57480b02f6\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") "
Nov 01 14:58:47.033648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.032927    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197dcbfb-9678-4806-bee1-4d57480b02f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197dcbfb-9678-4806-bee1-4d57480b02f6-openshift-service-ca\") pod \"197dcbfb-9678-4806-bee1-4d57480b02f6\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") "
Nov 01 14:58:47.033648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.033181    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197dcbfb-9678-4806-bee1-4d57480b02f6-kube-api-access-9wx2t\" (UniqueName: \"kubernetes.io/projected/197dcbfb-9678-4806-bee1-4d57480b02f6-kube-api-access-9wx2t\") pod \"197dcbfb-9678-4806-bee1-4d57480b02f6\" (UID: \"197dcbfb-9678-4806-bee1-4d57480b02f6\") "
Nov 01 14:58:47.033648 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.033547    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-maas-api-tls" pod "197dcbfb-9678-4806-bee1-4d57480b02f6" (UID: "197dcbfb-9678-4806-bee1-4d57480b02f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:58:47.033939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.033840    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:47.033939 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.033906    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197dcbfb-9678-4806-bee1-4d57480b02f6-openshift-service-ca" pod "197dcbfb-9678-4806-bee1-4d57480b02f6" (UID: "197dcbfb-9678-4806-bee1-4d57480b02f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:58:47.034568 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.034503    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-metrics-certs" pod "197dcbfb-9678-4806-bee1-4d57480b02f6" (UID: "197dcbfb-9678-4806-bee1-4d57480b02f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:58:47.048515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.048337    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197dcbfb-9678-4806-bee1-4d57480b02f6-kube-api-access-9wx2t" pod "197dcbfb-9678-4806-bee1-4d57480b02f6" (UID: "197dcbfb-9678-4806-bee1-4d57480b02f6"). InnerVolumeSpecName "kube-api-access-9wx2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:47.067882 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.067839    5423 generic.go:356] "Generic (PLEG): container finished" podID="197dcbfb-9678-4806-bee1-4d57480b02f6" containerID="49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791" exitCode=0
Nov 01 14:58:47.068067 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.067891    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv" event={"ID":"197dcbfb-9678-4806-bee1-4d57480b02f6","Type":"ContainerDied","Data":"49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791"}
Nov 01 14:58:47.068067 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.067965    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv" event={"ID":"197dcbfb-9678-4806-bee1-4d57480b02f6","Type":"ContainerDied","Data":"7fddd891a73b3dd754282756e9cfddee88e3aa3340efffd59528a241e2ae6e0d"}
Nov 01 14:58:47.068067 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.067997    5423 scope.go:122] "RemoveContainer" containerID="49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791"
Nov 01 14:58:47.068236 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.068135    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"
Nov 01 14:58:47.081550 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.081510    5423 scope.go:122] "RemoveContainer" containerID="49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791"
Nov 01 14:58:47.081819 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:58:47.081796    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791\": container with ID starting with 49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791 not found: ID does not exist" containerID="49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791"
Nov 01 14:58:47.081908 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.081836    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791"} err="failed to get container status \"49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791\": rpc error: code = NotFound desc = could not find container \"49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791\": container with ID starting with 49059549a56f1c475aa7c18ec3279c15c11ace7c44fed0d4fa5c5dce86721791 not found: ID does not exist"
Nov 01 14:58:47.134567 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.134540    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wx2t\" (UniqueName: \"kubernetes.io/projected/197dcbfb-9678-4806-bee1-4d57480b02f6-kube-api-access-9wx2t\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:47.134567 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.134565    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197dcbfb-9678-4806-bee1-4d57480b02f6-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:47.134713 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.134574    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197dcbfb-9678-4806-bee1-4d57480b02f6-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:47.245334 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.245250    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"]
Nov 01 14:58:47.248682 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.248659    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f764c7-b9777cdb9-99rgv"]
Nov 01 14:58:47.962251 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.962220    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197dcbfb-9678-4806-bee1-4d57480b02f6" path="/var/lib/kubelet/pods/197dcbfb-9678-4806-bee1-4d57480b02f6/volumes"
Nov 01 14:58:47.962788 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:47.962767    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a348208d-d63d-42f3-b681-b76d21d80fac" path="/var/lib/kubelet/pods/a348208d-d63d-42f3-b681-b76d21d80fac/volumes"
Nov 01 14:58:50.727697 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.727663    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"]
Nov 01 14:58:50.728374 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728353    5423 cpu_manager.go:414] "containerMap: removing container" podUID="8819031c-9890-49b6-8677-e37a34d67d25" containerName="payload-processing"
Nov 01 14:58:50.728496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728375    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8819031c-9890-49b6-8677-e37a34d67d25" containerName="payload-processing"
Nov 01 14:58:50.728496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728412    5423 cpu_manager.go:414] "containerMap: removing container" podUID="a348208d-d63d-42f3-b681-b76d21d80fac" containerName="maas-api"
Nov 01 14:58:50.728496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728421    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a348208d-d63d-42f3-b681-b76d21d80fac" containerName="maas-api"
Nov 01 14:58:50.728496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728431    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b7988724-6fe6-41d0-8f01-626458fe98f0" containerName="payload-pre-processing"
Nov 01 14:58:50.728496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728439    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7988724-6fe6-41d0-8f01-626458fe98f0" containerName="payload-pre-processing"
Nov 01 14:58:50.728496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728490    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b6dcd1d6-f381-4556-ba66-18197a1648a6" containerName="maas-api"
Nov 01 14:58:50.728496 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728498    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6dcd1d6-f381-4556-ba66-18197a1648a6" containerName="maas-api"
Nov 01 14:58:50.728847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728512    5423 cpu_manager.go:414] "containerMap: removing container" podUID="197dcbfb-9678-4806-bee1-4d57480b02f6" containerName="maas-api"
Nov 01 14:58:50.728847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728520    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197dcbfb-9678-4806-bee1-4d57480b02f6" containerName="maas-api"
Nov 01 14:58:50.728847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728645    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="8819031c-9890-49b6-8677-e37a34d67d25" containerName="payload-processing"
Nov 01 14:58:50.728847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728662    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="a348208d-d63d-42f3-b681-b76d21d80fac" containerName="maas-api"
Nov 01 14:58:50.728847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728672    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7988724-6fe6-41d0-8f01-626458fe98f0" containerName="payload-pre-processing"
Nov 01 14:58:50.728847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728687    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6dcd1d6-f381-4556-ba66-18197a1648a6" containerName="maas-api"
Nov 01 14:58:50.728847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.728696    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="197dcbfb-9678-4806-bee1-4d57480b02f6" containerName="maas-api"
Nov 01 14:58:50.733695 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.733668    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.785125 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.785063    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-data\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.785520 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.785498    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-credential-socket\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.785971 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.785886    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9pz\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-kube-api-access-7l9pz\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.786200 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.786182    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-socket\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.786460 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.786442    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-certs\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.786701 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.786684    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-podinfo\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.788226 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.788205    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4e07cb-openshift-default-dockercfg-94kfj\""
Nov 01 14:58:50.788452 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.788433    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-token\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.788732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.788714    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istiod-ca-cert\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.789017 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.789000    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-envoy\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.868377 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.868337    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"]
Nov 01 14:58:50.890230 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.890159    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-credential-socket\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.890525 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.890494    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9pz\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-kube-api-access-7l9pz\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.895540 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.895507    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-socket\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.917501    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-certs\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.917562    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-podinfo\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.917654    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-token\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.917759    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istiod-ca-cert\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.917977    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-envoy\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.918021    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-data\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.918501    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-data\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.894809    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-credential-socket\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.918785    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-certs\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.917402    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-socket\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.925408 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.919871    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istiod-ca-cert\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.942314 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.942017    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-envoy\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.960816 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.960641    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-podinfo\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.985780 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.985710    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-token\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:50.986248 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:50.986223    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9pz\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-kube-api-access-7l9pz\") pod \"e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") " pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:51.070533 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:51.070502    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:51.488817 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:58:51.488785    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb5f2ae_d498_4cde_95be_e7dd1339c347.slice/crio-fcd86eacf56071ace5a8799a0b6e90025a7a3bbafd0d57b7d6da631c21647ce4 WatchSource:0}: Error finding container fcd86eacf56071ace5a8799a0b6e90025a7a3bbafd0d57b7d6da631c21647ce4: Status 404 returned error can't find the container with id fcd86eacf56071ace5a8799a0b6e90025a7a3bbafd0d57b7d6da631c21647ce4
Nov 01 14:58:51.491367 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:51.491337    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:58:51.491485 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:51.491438    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:58:51.491485 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:51.491472    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:58:51.491658 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:51.491639    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"]
Nov 01 14:58:52.100917 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:52.100882    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" event={"ID":"bbb5f2ae-d498-4cde-95be-e7dd1339c347","Type":"ContainerStarted","Data":"609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32"}
Nov 01 14:58:52.101357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:52.100938    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" event={"ID":"bbb5f2ae-d498-4cde-95be-e7dd1339c347","Type":"ContainerStarted","Data":"fcd86eacf56071ace5a8799a0b6e90025a7a3bbafd0d57b7d6da631c21647ce4"}
Nov 01 14:58:52.222020 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:52.221970    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podStartSLOduration=2.221957263 podStartE2EDuration="2.221957263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:52.218768234 +0000 UTC m=+2472.384253532" watchObservedRunningTime="2026-11-01 14:58:52.221957263 +0000 UTC m=+2472.387442562"
Nov 01 14:58:53.071161 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.071126    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:53.076706 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.076679    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:53.106915 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.106881    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:53.107782 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.107760    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:58:53.341466 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.341445    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:58:53.451599 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.451571    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbed19b3-8946-4aa8-81b0-8d5874180d70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbed19b3-8946-4aa8-81b0-8d5874180d70-plugins-config-volume\") pod \"cbed19b3-8946-4aa8-81b0-8d5874180d70\" (UID: \"cbed19b3-8946-4aa8-81b0-8d5874180d70\") "
Nov 01 14:58:53.451767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.451655    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbed19b3-8946-4aa8-81b0-8d5874180d70-kube-api-access-f8cl2\" (UniqueName: \"kubernetes.io/projected/cbed19b3-8946-4aa8-81b0-8d5874180d70-kube-api-access-f8cl2\") pod \"cbed19b3-8946-4aa8-81b0-8d5874180d70\" (UID: \"cbed19b3-8946-4aa8-81b0-8d5874180d70\") "
Nov 01 14:58:53.451937 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.451908    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbed19b3-8946-4aa8-81b0-8d5874180d70-plugins-config-volume" pod "cbed19b3-8946-4aa8-81b0-8d5874180d70" (UID: "cbed19b3-8946-4aa8-81b0-8d5874180d70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:58:53.453739 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.453710    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbed19b3-8946-4aa8-81b0-8d5874180d70-kube-api-access-f8cl2" pod "cbed19b3-8946-4aa8-81b0-8d5874180d70" (UID: "cbed19b3-8946-4aa8-81b0-8d5874180d70"). InnerVolumeSpecName "kube-api-access-f8cl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:53.553220 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.553183    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbed19b3-8946-4aa8-81b0-8d5874180d70-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:53.553220 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:53.553214    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8cl2\" (UniqueName: \"kubernetes.io/projected/cbed19b3-8946-4aa8-81b0-8d5874180d70-kube-api-access-f8cl2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:54.114166 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.114133    5423 generic.go:356] "Generic (PLEG): container finished" podID="cbed19b3-8946-4aa8-81b0-8d5874180d70" containerID="ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a" exitCode=0
Nov 01 14:58:54.114657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.114197    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"
Nov 01 14:58:54.114657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.114221    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz" event={"ID":"cbed19b3-8946-4aa8-81b0-8d5874180d70","Type":"ContainerDied","Data":"ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a"}
Nov 01 14:58:54.114657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.114289    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz" event={"ID":"cbed19b3-8946-4aa8-81b0-8d5874180d70","Type":"ContainerDied","Data":"5c8fff4a4156f36c1b73e890bd0736e2525313836eec8cd7a7df948c6b730df3"}
Nov 01 14:58:54.114657 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.114325    5423 scope.go:122] "RemoveContainer" containerID="ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a"
Nov 01 14:58:54.127007 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.126988    5423 scope.go:122] "RemoveContainer" containerID="ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a"
Nov 01 14:58:54.127297 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:58:54.127275    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a\": container with ID starting with ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a not found: ID does not exist" containerID="ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a"
Nov 01 14:58:54.127379 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.127307    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a"} err="failed to get container status \"ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a\": rpc error: code = NotFound desc = could not find container \"ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a\": container with ID starting with ec9c227e50213cc544dad447b2fa69234bc21a2b1ef8aa2d0bdab1f378026f6a not found: ID does not exist"
Nov 01 14:58:54.283669 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.283598    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"]
Nov 01 14:58:54.334334 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:54.334294    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f764c7-6c977c4bcc-cbrvz"]
Nov 01 14:58:55.963776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:58:55.963736    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbed19b3-8946-4aa8-81b0-8d5874180d70" path="/var/lib/kubelet/pods/cbed19b3-8946-4aa8-81b0-8d5874180d70/volumes"
Nov 01 14:59:02.511183 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:02.511147    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"]
Nov 01 14:59:02.511764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:02.511370    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w" podUID="b8e22105-74fd-42bc-90bc-103772edf87a" containerName="payload-processing" containerID="cri-o://f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96" gracePeriod=30
Nov 01 14:59:02.562347 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:02.562314    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"]
Nov 01 14:59:02.562601 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:02.562559    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh" podUID="4d9d5278-63b2-4148-9cfa-9ce46f31bf6f" containerName="payload-pre-processing" containerID="cri-o://4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228" gracePeriod=30
Nov 01 14:59:02.879219 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:02.879198    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:59:02.882249 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:02.882229    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:59:03.042255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.042173    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e22105-74fd-42bc-90bc-103772edf87a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e22105-74fd-42bc-90bc-103772edf87a-plugins-config-volume\") pod \"b8e22105-74fd-42bc-90bc-103772edf87a\" (UID: \"b8e22105-74fd-42bc-90bc-103772edf87a\") "
Nov 01 14:59:03.042452 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.042331    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-kube-api-access-hhl7h\" (UniqueName: \"kubernetes.io/projected/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-kube-api-access-hhl7h\") pod \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\" (UID: \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\") "
Nov 01 14:59:03.042452 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.042376    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e22105-74fd-42bc-90bc-103772edf87a-kube-api-access-67dfd\" (UniqueName: \"kubernetes.io/projected/b8e22105-74fd-42bc-90bc-103772edf87a-kube-api-access-67dfd\") pod \"b8e22105-74fd-42bc-90bc-103772edf87a\" (UID: \"b8e22105-74fd-42bc-90bc-103772edf87a\") "
Nov 01 14:59:03.042571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.042458    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-plugins-config-volume\") pod \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\" (UID: \"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f\") "
Nov 01 14:59:03.042625 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.042601    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e22105-74fd-42bc-90bc-103772edf87a-plugins-config-volume" pod "b8e22105-74fd-42bc-90bc-103772edf87a" (UID: "b8e22105-74fd-42bc-90bc-103772edf87a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:59:03.042994 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.042975    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e22105-74fd-42bc-90bc-103772edf87a-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:03.043376 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.043352    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-plugins-config-volume" pod "4d9d5278-63b2-4148-9cfa-9ce46f31bf6f" (UID: "4d9d5278-63b2-4148-9cfa-9ce46f31bf6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:59:03.044755 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.044731    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-kube-api-access-hhl7h" pod "4d9d5278-63b2-4148-9cfa-9ce46f31bf6f" (UID: "4d9d5278-63b2-4148-9cfa-9ce46f31bf6f"). InnerVolumeSpecName "kube-api-access-hhl7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:59:03.044867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.044833    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e22105-74fd-42bc-90bc-103772edf87a-kube-api-access-67dfd" pod "b8e22105-74fd-42bc-90bc-103772edf87a" (UID: "b8e22105-74fd-42bc-90bc-103772edf87a"). InnerVolumeSpecName "kube-api-access-67dfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:59:03.171776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.144629    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhl7h\" (UniqueName: \"kubernetes.io/projected/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-kube-api-access-hhl7h\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:03.171776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.144665    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67dfd\" (UniqueName: \"kubernetes.io/projected/b8e22105-74fd-42bc-90bc-103772edf87a-kube-api-access-67dfd\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:03.171776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.144680    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:03.226409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.215572    5423 generic.go:356] "Generic (PLEG): container finished" podID="b8e22105-74fd-42bc-90bc-103772edf87a" containerID="f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96" exitCode=0
Nov 01 14:59:03.226409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.215677    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w" event={"ID":"b8e22105-74fd-42bc-90bc-103772edf87a","Type":"ContainerDied","Data":"f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96"}
Nov 01 14:59:03.226409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.215730    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w" event={"ID":"b8e22105-74fd-42bc-90bc-103772edf87a","Type":"ContainerDied","Data":"bbb13d1f5a92a9bc1cbfe0b007348b7b73adf588bb35d151a01c5f5621795f81"}
Nov 01 14:59:03.226409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.215761    5423 scope.go:122] "RemoveContainer" containerID="f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96"
Nov 01 14:59:03.226409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.215933    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"
Nov 01 14:59:03.249660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.240129    5423 generic.go:356] "Generic (PLEG): container finished" podID="4d9d5278-63b2-4148-9cfa-9ce46f31bf6f" containerID="4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228" exitCode=0
Nov 01 14:59:03.249660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.240274    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh" event={"ID":"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f","Type":"ContainerDied","Data":"4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228"}
Nov 01 14:59:03.249660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.240319    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh" event={"ID":"4d9d5278-63b2-4148-9cfa-9ce46f31bf6f","Type":"ContainerDied","Data":"4c2af58972ac4579b8c87ee78d5594d035566e7d59617d13aa365e5cd0b72415"}
Nov 01 14:59:03.249660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.241872    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"
Nov 01 14:59:03.269661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.259061    5423 scope.go:122] "RemoveContainer" containerID="f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96"
Nov 01 14:59:03.269661 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:03.259407    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96\": container with ID starting with f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96 not found: ID does not exist" containerID="f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96"
Nov 01 14:59:03.269661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.259445    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96"} err="failed to get container status \"f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96\": rpc error: code = NotFound desc = could not find container \"f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96\": container with ID starting with f550313301b8dc9bf3ba7f771eaa77e28ddb27a1e846d6cb8eda07ed4d411a96 not found: ID does not exist"
Nov 01 14:59:03.269661 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.259470    5423 scope.go:122] "RemoveContainer" containerID="4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228"
Nov 01 14:59:03.287432 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.281823    5423 scope.go:122] "RemoveContainer" containerID="4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228"
Nov 01 14:59:03.287432 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:03.282328    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228\": container with ID starting with 4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228 not found: ID does not exist" containerID="4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228"
Nov 01 14:59:03.287432 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.282362    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228"} err="failed to get container status \"4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228\": rpc error: code = NotFound desc = could not find container \"4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228\": container with ID starting with 4c60c2af12312dc2daa18e02d5a527ab691289c3df68150d3d9fa8a29d503228 not found: ID does not exist"
Nov 01 14:59:03.429783 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.429748    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"]
Nov 01 14:59:03.433598 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.433573    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-f85689cb4-mp94w"]
Nov 01 14:59:03.548653 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.548618    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"]
Nov 01 14:59:03.552841 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.552812    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-779f96959f-gdpdh"]
Nov 01 14:59:03.966050 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.966009    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9d5278-63b2-4148-9cfa-9ce46f31bf6f" path="/var/lib/kubelet/pods/4d9d5278-63b2-4148-9cfa-9ce46f31bf6f/volumes"
Nov 01 14:59:03.966590 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:03.966571    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e22105-74fd-42bc-90bc-103772edf87a" path="/var/lib/kubelet/pods/b8e22105-74fd-42bc-90bc-103772edf87a/volumes"
Nov 01 14:59:05.245771 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:05.245737    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:59:05.888531 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:05.888501    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:59:07.260125 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:07.260072    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"]
Nov 01 14:59:07.262473 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:07.260321    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy" containerID="cri-o://f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e" gracePeriod=30
Nov 01 14:59:08.018651 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.018608    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"]
Nov 01 14:59:08.030634 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.030588    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b8e22105-74fd-42bc-90bc-103772edf87a" containerName="payload-processing"
Nov 01 14:59:08.030879 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.030851    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e22105-74fd-42bc-90bc-103772edf87a" containerName="payload-processing"
Nov 01 14:59:08.031034 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.031022    5423 cpu_manager.go:414] "containerMap: removing container" podUID="cbed19b3-8946-4aa8-81b0-8d5874180d70" containerName="payload-processing"
Nov 01 14:59:08.031162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.031149    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbed19b3-8946-4aa8-81b0-8d5874180d70" containerName="payload-processing"
Nov 01 14:59:08.031306 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.031276    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4d9d5278-63b2-4148-9cfa-9ce46f31bf6f" containerName="payload-pre-processing"
Nov 01 14:59:08.031437 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.031422    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9d5278-63b2-4148-9cfa-9ce46f31bf6f" containerName="payload-pre-processing"
Nov 01 14:59:08.031815 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.031796    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e22105-74fd-42bc-90bc-103772edf87a" containerName="payload-processing"
Nov 01 14:59:08.031948 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.031936    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbed19b3-8946-4aa8-81b0-8d5874180d70" containerName="payload-processing"
Nov 01 14:59:08.032150 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.032046    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9d5278-63b2-4148-9cfa-9ce46f31bf6f" containerName="payload-pre-processing"
Nov 01 14:59:08.185707 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.185509    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:08.291379 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.291296    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkqj\" (UniqueName: \"kubernetes.io/projected/4c209a58-3ede-47f3-81dd-813a0ba66507-kube-api-access-rkkqj\") pod \"payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv\" (UID: \"4c209a58-3ede-47f3-81dd-813a0ba66507\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:08.292064 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.292037    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c209a58-3ede-47f3-81dd-813a0ba66507-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv\" (UID: \"4c209a58-3ede-47f3-81dd-813a0ba66507\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:08.335406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.328054    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-97c87c-dockercfg-bbd2m\""
Nov 01 14:59:08.362252 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.343782    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-97c87c\""
Nov 01 14:59:08.397109 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.396987    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkqj\" (UniqueName: \"kubernetes.io/projected/4c209a58-3ede-47f3-81dd-813a0ba66507-kube-api-access-rkkqj\") pod \"payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv\" (UID: \"4c209a58-3ede-47f3-81dd-813a0ba66507\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:08.412777 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.412745    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c209a58-3ede-47f3-81dd-813a0ba66507-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv\" (UID: \"4c209a58-3ede-47f3-81dd-813a0ba66507\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:08.413180 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.413157    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c209a58-3ede-47f3-81dd-813a0ba66507-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv\" (UID: \"4c209a58-3ede-47f3-81dd-813a0ba66507\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:08.661744 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.661718    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"]
Nov 01 14:59:08.664425 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.664364    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkqj\" (UniqueName: \"kubernetes.io/projected/4c209a58-3ede-47f3-81dd-813a0ba66507-kube-api-access-rkkqj\") pod \"payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv\" (UID: \"4c209a58-3ede-47f3-81dd-813a0ba66507\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:08.958032 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:08.957955    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:09.345444 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:59:09.345415    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c209a58_3ede_47f3_81dd_813a0ba66507.slice/crio-b5ec046cb7302bd23d422c6093fe0db56465c1982eea6445412ae63c8e3ee9a1 WatchSource:0}: Error finding container b5ec046cb7302bd23d422c6093fe0db56465c1982eea6445412ae63c8e3ee9a1: Status 404 returned error can't find the container with id b5ec046cb7302bd23d422c6093fe0db56465c1982eea6445412ae63c8e3ee9a1
Nov 01 14:59:09.346746 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:09.346724    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"]
Nov 01 14:59:09.667999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:09.667883    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv" event={"ID":"4c209a58-3ede-47f3-81dd-813a0ba66507","Type":"ContainerStarted","Data":"612a176f3c39252dd6135fb0641512a4105dab27d31624ce5bde6c187d0f31ef"}
Nov 01 14:59:09.667999 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:09.667971    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv" event={"ID":"4c209a58-3ede-47f3-81dd-813a0ba66507","Type":"ContainerStarted","Data":"b5ec046cb7302bd23d422c6093fe0db56465c1982eea6445412ae63c8e3ee9a1"}
Nov 01 14:59:09.668243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:09.668014    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:09.790409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:09.789771    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv" podStartSLOduration=2.789751229 podStartE2EDuration="2.789751229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:09.786977764 +0000 UTC m=+2489.952463063" watchObservedRunningTime="2026-11-01 14:59:09.789751229 +0000 UTC m=+2489.955236528"
Nov 01 14:59:12.537740 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.537680    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:59:12.580206 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580181    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-certs\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580237    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-data\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580285    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-socket\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580349    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istiod-ca-cert\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580416    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-podinfo\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580482    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-token\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-token\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580478    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-certs" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:12.580571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580540    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-envoy\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580571 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580550    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-data" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:12.580823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580576    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-kube-api-access-q4rw5\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-kube-api-access-q4rw5\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580644    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-credential-socket\") pod \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\" (UID: \"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c\") "
Nov 01 14:59:12.580823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580660    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-socket" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:12.580823 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580782    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istiod-ca-cert" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:59:12.581010 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.580906    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-credential-socket" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:12.581243 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.581221    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.581366 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.581246    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.581366 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.581260    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.581366 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.581273    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.581366 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.581285    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.582847 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.582825    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-podinfo" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:59:12.583042 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.583021    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-token" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:59:12.583274 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.583249    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-kube-api-access-q4rw5" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "kube-api-access-q4rw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:59:12.583349 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.583319    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-envoy" pod "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" (UID: "b2fea37e-0d2a-46ac-8c9d-cbe688baa79c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:12.682130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.682096    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.682130 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.682126    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.682311 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.682141    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.682311 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.682154    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4rw5\" (UniqueName: \"kubernetes.io/projected/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c-kube-api-access-q4rw5\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:12.687124 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.687092    5423 generic.go:356] "Generic (PLEG): container finished" podID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerID="f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e" exitCode=0
Nov 01 14:59:12.687295 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.687133    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" event={"ID":"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c","Type":"ContainerDied","Data":"f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e"}
Nov 01 14:59:12.687378 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.687169    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"
Nov 01 14:59:12.687471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.687409    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h" event={"ID":"b2fea37e-0d2a-46ac-8c9d-cbe688baa79c","Type":"ContainerDied","Data":"eaacd69933924f9b3fa82e47d6646afd457d6e76c30ac4cd30bc9ee1b0e29be5"}
Nov 01 14:59:12.687471 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.687453    5423 scope.go:122] "RemoveContainer" containerID="f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e"
Nov 01 14:59:12.701307 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.701290    5423 scope.go:122] "RemoveContainer" containerID="f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e"
Nov 01 14:59:12.701828 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:12.701807    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e\": container with ID starting with f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e not found: ID does not exist" containerID="f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e"
Nov 01 14:59:12.701899 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.701836    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e"} err="failed to get container status \"f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e\": rpc error: code = NotFound desc = could not find container \"f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e\": container with ID starting with f1cea90ac9ea45915265aabc89ba5543fe24d81febc10eec1d1fa0297bdf872e not found: ID does not exist"
Nov 01 14:59:12.829143 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.829106    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"]
Nov 01 14:59:12.836406 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:12.832184    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-52j2h"]
Nov 01 14:59:13.961044 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:13.961015    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" path="/var/lib/kubelet/pods/b2fea37e-0d2a-46ac-8c9d-cbe688baa79c/volumes"
Nov 01 14:59:16.045779 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.045748    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"]
Nov 01 14:59:16.046263 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.046242    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy"
Nov 01 14:59:16.046263 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.046254    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy"
Nov 01 14:59:16.046381 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.046367    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2fea37e-0d2a-46ac-8c9d-cbe688baa79c" containerName="istio-proxy"
Nov 01 14:59:16.047255 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.047222    5423 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","poda348208d-d63d-42f3-b681-b76d21d80fac"] err="unable to destroy cgroup paths for cgroup [kubepods burstable poda348208d-d63d-42f3-b681-b76d21d80fac] : Timed out while waiting for systemd to remove kubepods-burstable-poda348208d_d63d_42f3_b681_b76d21d80fac.slice"
Nov 01 14:59:16.047361 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.047256    5423 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="a348208d-d63d-42f3-b681-b76d21d80fac" err="failed to delete cgroup paths for [kubepods burstable poda348208d-d63d-42f3-b681-b76d21d80fac] : unable to destroy cgroup paths for cgroup [kubepods burstable poda348208d-d63d-42f3-b681-b76d21d80fac] : Timed out while waiting for systemd to remove kubepods-burstable-poda348208d_d63d_42f3_b681_b76d21d80fac.slice"
Nov 01 14:59:16.051397 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.051365    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:16.108132 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.108098    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"]
Nov 01 14:59:16.137414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.133111    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lqm\" (UniqueName: \"kubernetes.io/projected/1e75b990-47d1-49a6-ae96-78ce34057475-kube-api-access-s7lqm\") pod \"payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr\" (UID: \"1e75b990-47d1-49a6-ae96-78ce34057475\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:16.137414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.133382    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e75b990-47d1-49a6-ae96-78ce34057475-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr\" (UID: \"1e75b990-47d1-49a6-ae96-78ce34057475\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:16.175409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.174694    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-d06633-dockercfg-8swvw\""
Nov 01 14:59:16.175409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.174805    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-d06633\""
Nov 01 14:59:16.239409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.238762    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e75b990-47d1-49a6-ae96-78ce34057475-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr\" (UID: \"1e75b990-47d1-49a6-ae96-78ce34057475\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:16.239409 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.238986    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lqm\" (UniqueName: \"kubernetes.io/projected/1e75b990-47d1-49a6-ae96-78ce34057475-kube-api-access-s7lqm\") pod \"payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr\" (UID: \"1e75b990-47d1-49a6-ae96-78ce34057475\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:16.247259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.240377    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e75b990-47d1-49a6-ae96-78ce34057475-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr\" (UID: \"1e75b990-47d1-49a6-ae96-78ce34057475\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:16.252791 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.252764    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"]
Nov 01 14:59:16.263450 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.261241    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:16.296537 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.296477    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"]
Nov 01 14:59:16.299008 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.298984    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lqm\" (UniqueName: \"kubernetes.io/projected/1e75b990-47d1-49a6-ae96-78ce34057475-kube-api-access-s7lqm\") pod \"payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr\" (UID: \"1e75b990-47d1-49a6-ae96-78ce34057475\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:16.340357 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.340311    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84886640-64b4-4cc6-b63d-fbaee23c54bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk\" (UID: \"84886640-64b4-4cc6-b63d-fbaee23c54bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:16.340551 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.340440    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcwjn\" (UniqueName: \"kubernetes.io/projected/84886640-64b4-4cc6-b63d-fbaee23c54bd-kube-api-access-rcwjn\") pod \"payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk\" (UID: \"84886640-64b4-4cc6-b63d-fbaee23c54bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:16.378812 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.378781    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:16.446995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.441970    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84886640-64b4-4cc6-b63d-fbaee23c54bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk\" (UID: \"84886640-64b4-4cc6-b63d-fbaee23c54bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:16.446995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.442051    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcwjn\" (UniqueName: \"kubernetes.io/projected/84886640-64b4-4cc6-b63d-fbaee23c54bd-kube-api-access-rcwjn\") pod \"payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk\" (UID: \"84886640-64b4-4cc6-b63d-fbaee23c54bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:16.446995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.444082    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84886640-64b4-4cc6-b63d-fbaee23c54bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk\" (UID: \"84886640-64b4-4cc6-b63d-fbaee23c54bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:16.503650 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.503615    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcwjn\" (UniqueName: \"kubernetes.io/projected/84886640-64b4-4cc6-b63d-fbaee23c54bd-kube-api-access-rcwjn\") pod \"payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk\" (UID: \"84886640-64b4-4cc6-b63d-fbaee23c54bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:16.589133 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.589108    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:16.745280 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:59:16.745237    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e75b990_47d1_49a6_ae96_78ce34057475.slice/crio-e7c6bd90ef0583d15641362450cfee7b579d2c9dcedbc969a92c26d041e0dfd3 WatchSource:0}: Error finding container e7c6bd90ef0583d15641362450cfee7b579d2c9dcedbc969a92c26d041e0dfd3: Status 404 returned error can't find the container with id e7c6bd90ef0583d15641362450cfee7b579d2c9dcedbc969a92c26d041e0dfd3
Nov 01 14:59:16.748116 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.748085    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"]
Nov 01 14:59:16.959839 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:16.959817    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"]
Nov 01 14:59:17.718244 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:17.718205    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk" event={"ID":"84886640-64b4-4cc6-b63d-fbaee23c54bd","Type":"ContainerStarted","Data":"e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec"}
Nov 01 14:59:17.718734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:17.718283    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk" event={"ID":"84886640-64b4-4cc6-b63d-fbaee23c54bd","Type":"ContainerStarted","Data":"7e9d12540eb66e005081d063d6165c571e3370d46aaecf87d9424ab4c8d3a9cf"}
Nov 01 14:59:17.718734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:17.718327    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:17.719845 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:17.719825    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr" event={"ID":"1e75b990-47d1-49a6-ae96-78ce34057475","Type":"ContainerStarted","Data":"70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9"}
Nov 01 14:59:17.719960 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:17.719857    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr" event={"ID":"1e75b990-47d1-49a6-ae96-78ce34057475","Type":"ContainerStarted","Data":"e7c6bd90ef0583d15641362450cfee7b579d2c9dcedbc969a92c26d041e0dfd3"}
Nov 01 14:59:17.719997 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:17.719956    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:17.836591 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:17.836540    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk" podStartSLOduration=1.836528878 podStartE2EDuration="1.836528878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:17.835152556 +0000 UTC m=+2498.000637854" watchObservedRunningTime="2026-11-01 14:59:17.836528878 +0000 UTC m=+2498.002014176"
Nov 01 14:59:17.965018 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:17.964954    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr" podStartSLOduration=1.964933468 podStartE2EDuration="1.964933468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:17.962844749 +0000 UTC m=+2498.128330048" watchObservedRunningTime="2026-11-01 14:59:17.964933468 +0000 UTC m=+2498.130418768"
Nov 01 14:59:20.551478 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.551442    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"]
Nov 01 14:59:20.555510 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.555488    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:20.612052 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.612010    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"]
Nov 01 14:59:20.694583 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.670667    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0c9d0a\""
Nov 01 14:59:20.694583 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.670820    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0c9d0a-dockercfg-42llx\""
Nov 01 14:59:20.747262 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.747206    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7kgj\" (UniqueName: \"kubernetes.io/projected/84836356-3ad8-4aca-bea9-ff472a135724-kube-api-access-g7kgj\") pod \"payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp\" (UID: \"84836356-3ad8-4aca-bea9-ff472a135724\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:20.747562 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.747282    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84836356-3ad8-4aca-bea9-ff472a135724-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp\" (UID: \"84836356-3ad8-4aca-bea9-ff472a135724\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:20.748395 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.748350    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"]
Nov 01 14:59:20.764811 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.764773    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:20.768506 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.768474    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 14:59:20.780906 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.780874    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"]
Nov 01 14:59:20.848379 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.848334    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7kgj\" (UniqueName: \"kubernetes.io/projected/84836356-3ad8-4aca-bea9-ff472a135724-kube-api-access-g7kgj\") pod \"payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp\" (UID: \"84836356-3ad8-4aca-bea9-ff472a135724\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:20.848556 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.848418    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84836356-3ad8-4aca-bea9-ff472a135724-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp\" (UID: \"84836356-3ad8-4aca-bea9-ff472a135724\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:20.849100 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.849081    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84836356-3ad8-4aca-bea9-ff472a135724-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp\" (UID: \"84836356-3ad8-4aca-bea9-ff472a135724\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:20.905929 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.905902    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7kgj\" (UniqueName: \"kubernetes.io/projected/84836356-3ad8-4aca-bea9-ff472a135724-kube-api-access-g7kgj\") pod \"payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp\" (UID: \"84836356-3ad8-4aca-bea9-ff472a135724\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:20.949482 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.949457    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b4a585-907c-4d1f-a073-31087845f7e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk\" (UID: \"86b4a585-907c-4d1f-a073-31087845f7e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:20.949595 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:20.949573    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpz2f\" (UniqueName: \"kubernetes.io/projected/86b4a585-907c-4d1f-a073-31087845f7e6-kube-api-access-vpz2f\") pod \"payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk\" (UID: \"86b4a585-907c-4d1f-a073-31087845f7e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:21.051002 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.050969    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b4a585-907c-4d1f-a073-31087845f7e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk\" (UID: \"86b4a585-907c-4d1f-a073-31087845f7e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:21.051157 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.051050    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpz2f\" (UniqueName: \"kubernetes.io/projected/86b4a585-907c-4d1f-a073-31087845f7e6-kube-api-access-vpz2f\") pod \"payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk\" (UID: \"86b4a585-907c-4d1f-a073-31087845f7e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:21.051720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.051702    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b4a585-907c-4d1f-a073-31087845f7e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk\" (UID: \"86b4a585-907c-4d1f-a073-31087845f7e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:21.116121 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.116043    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpz2f\" (UniqueName: \"kubernetes.io/projected/86b4a585-907c-4d1f-a073-31087845f7e6-kube-api-access-vpz2f\") pod \"payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk\" (UID: \"86b4a585-907c-4d1f-a073-31087845f7e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:21.170339 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.170307    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:21.385404 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.385306    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:21.539858 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:59:21.539821    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84836356_3ad8_4aca_bea9_ff472a135724.slice/crio-87cc1fb13e45da9ae9efb62f71d940c8704fc806e7844ad9ca3cbd598f4d636a WatchSource:0}: Error finding container 87cc1fb13e45da9ae9efb62f71d940c8704fc806e7844ad9ca3cbd598f4d636a: Status 404 returned error can't find the container with id 87cc1fb13e45da9ae9efb62f71d940c8704fc806e7844ad9ca3cbd598f4d636a
Nov 01 14:59:21.543414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.543377    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"]
Nov 01 14:59:21.762125 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:59:21.762094    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b4a585_907c_4d1f_a073_31087845f7e6.slice/crio-89a554df7f31faf7d74c831d13c8644a6c0b4a110b90e60399afd7b833c4077c WatchSource:0}: Error finding container 89a554df7f31faf7d74c831d13c8644a6c0b4a110b90e60399afd7b833c4077c: Status 404 returned error can't find the container with id 89a554df7f31faf7d74c831d13c8644a6c0b4a110b90e60399afd7b833c4077c
Nov 01 14:59:21.763506 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.763482    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"]
Nov 01 14:59:21.768924 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.768897    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp" event={"ID":"84836356-3ad8-4aca-bea9-ff472a135724","Type":"ContainerStarted","Data":"1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e"}
Nov 01 14:59:21.769026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.768946    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp" event={"ID":"84836356-3ad8-4aca-bea9-ff472a135724","Type":"ContainerStarted","Data":"87cc1fb13e45da9ae9efb62f71d940c8704fc806e7844ad9ca3cbd598f4d636a"}
Nov 01 14:59:21.769026 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.769001    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:21.885917 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:21.885863    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp" podStartSLOduration=1.885849085 podStartE2EDuration="1.885849085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:21.88437444 +0000 UTC m=+2502.049859738" watchObservedRunningTime="2026-11-01 14:59:21.885849085 +0000 UTC m=+2502.051334431"
Nov 01 14:59:22.776345 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:22.776299    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk" event={"ID":"86b4a585-907c-4d1f-a073-31087845f7e6","Type":"ContainerStarted","Data":"764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe"}
Nov 01 14:59:22.776764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:22.776356    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk" event={"ID":"86b4a585-907c-4d1f-a073-31087845f7e6","Type":"ContainerStarted","Data":"89a554df7f31faf7d74c831d13c8644a6c0b4a110b90e60399afd7b833c4077c"}
Nov 01 14:59:22.776764 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:22.776377    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:22.894868 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:22.894807    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk" podStartSLOduration=2.8947937660000003 podStartE2EDuration="2.894793766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:22.893114617 +0000 UTC m=+2503.058599938" watchObservedRunningTime="2026-11-01 14:59:22.894793766 +0000 UTC m=+2503.060279108"
Nov 01 14:59:27.080162 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.080122    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"]
Nov 01 14:59:27.080597 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.080325    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm" podUID="e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" containerName="maas-api" containerID="cri-o://d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067" gracePeriod=30
Nov 01 14:59:27.399119 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.399090    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:59:27.462685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.461711    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-openshift-service-ca\") pod \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") "
Nov 01 14:59:27.462685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.461774    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls\") pod \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") "
Nov 01 14:59:27.462685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.461990    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-metrics-certs\") pod \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") "
Nov 01 14:59:27.462685 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.462030    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-kube-api-access-r87bj\" (UniqueName: \"kubernetes.io/projected/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-kube-api-access-r87bj\") pod \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\" (UID: \"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606\") "
Nov 01 14:59:27.487261 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.487207    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-openshift-service-ca" pod "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" (UID: "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:59:27.503021 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.502966    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-kube-api-access-r87bj" pod "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" (UID: "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606"). InnerVolumeSpecName "kube-api-access-r87bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:59:27.511962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.511900    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-metrics-certs" pod "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" (UID: "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:59:27.511962 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.511941    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls" pod "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" (UID: "e00ee4b7-5cf6-47a2-ad77-1c0642ba0606"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:59:27.563246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.563214    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:27.563246 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.563245    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r87bj\" (UniqueName: \"kubernetes.io/projected/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-kube-api-access-r87bj\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:27.563465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.563258    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:27.563465 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.563271    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:27.808150 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.808113    5423 generic.go:356] "Generic (PLEG): container finished" podID="e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" containerID="d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067" exitCode=0
Nov 01 14:59:27.808355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.808254    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm" event={"ID":"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606","Type":"ContainerDied","Data":"d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067"}
Nov 01 14:59:27.808355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.808309    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm" event={"ID":"e00ee4b7-5cf6-47a2-ad77-1c0642ba0606","Type":"ContainerDied","Data":"083f10daff55eecd6bc9ccaa4b66d61819243cb6f9a383768a62920c55129d06"}
Nov 01 14:59:27.808355 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.808341    5423 scope.go:122] "RemoveContainer" containerID="d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067"
Nov 01 14:59:27.808546 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.808521    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"
Nov 01 14:59:27.819313 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.819293    5423 scope.go:122] "RemoveContainer" containerID="d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067"
Nov 01 14:59:27.819595 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:27.819563    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067\": container with ID starting with d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067 not found: ID does not exist" containerID="d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067"
Nov 01 14:59:27.819699 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:27.819604    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067"} err="failed to get container status \"d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067\": rpc error: code = NotFound desc = could not find container \"d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067\": container with ID starting with d02251b88025b4126f4bdf4342629d2da0d91746eb064e97b4c7c84161c44067 not found: ID does not exist"
Nov 01 14:59:28.015542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.015499    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"]
Nov 01 14:59:28.037902 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.037870    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5223a8-666dc6b95b-cbwlm"]
Nov 01 14:59:28.570177 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.570135    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"]
Nov 01 14:59:28.570993 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.570964    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" containerName="maas-api"
Nov 01 14:59:28.571184 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.571159    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" containerName="maas-api"
Nov 01 14:59:28.571412 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.571381    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" containerName="maas-api"
Nov 01 14:59:28.596414 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.593301    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.636542 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.636497    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"]
Nov 01 14:59:28.694405 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.694363    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4e07cb-serving-cert\""
Nov 01 14:59:28.728774 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.728736    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.729047 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.729025    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.729229 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.729201    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrgl\" (UniqueName: \"kubernetes.io/projected/354e828e-9394-4e85-894b-f2930448c67d-kube-api-access-lrrgl\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.729510 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.729467    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354e828e-9394-4e85-894b-f2930448c67d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.729862 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.729843    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 14:59:28.729928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.729918    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 14:59:28.830879 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.830784    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.830879 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.830836    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.831120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.830905    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrgl\" (UniqueName: \"kubernetes.io/projected/354e828e-9394-4e85-894b-f2930448c67d-kube-api-access-lrrgl\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.831120 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.831069    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354e828e-9394-4e85-894b-f2930448c67d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.831734 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.831713    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354e828e-9394-4e85-894b-f2930448c67d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.833280 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.833250    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.833401 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.833294    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.889799 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.889758    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrgl\" (UniqueName: \"kubernetes.io/projected/354e828e-9394-4e85-894b-f2930448c67d-kube-api-access-lrrgl\") pod \"maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:28.957338 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:28.957308    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:29.556419 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:29.553346    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"]
Nov 01 14:59:29.853011 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:29.852970    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g" event={"ID":"354e828e-9394-4e85-894b-f2930448c67d","Type":"ContainerStarted","Data":"3a6d4df6596eae0e0c0243b10276d0ac3cf1364417868e91733de84f63772dbe"}
Nov 01 14:59:29.986721 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:29.986680    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00ee4b7-5cf6-47a2-ad77-1c0642ba0606" path="/var/lib/kubelet/pods/e00ee4b7-5cf6-47a2-ad77-1c0642ba0606/volumes"
Nov 01 14:59:30.860537 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:30.860497    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g" event={"ID":"354e828e-9394-4e85-894b-f2930448c67d","Type":"ContainerStarted","Data":"870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b"}
Nov 01 14:59:30.860916 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:30.860564    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:30.979225 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:30.979175    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g" podStartSLOduration=2.030839744 podStartE2EDuration="2.979161306s" totalImagesPullingTime="948.321562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:30.977598635 +0000 UTC m=+2511.143083932" watchObservedRunningTime="2026-11-01 14:59:30.979161306 +0000 UTC m=+2511.144646603"
Nov 01 14:59:32.779601 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:32.779568    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 14:59:33.786683 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:33.786655    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 14:59:35.148961 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:35.148927    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 14:59:36.871809 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:36.871781    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 14:59:38.045299 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.045262    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"]
Nov 01 14:59:38.049472 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.049449    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.098694 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.098662    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.098867 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.098821    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.098952 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.098885    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvb69\" (UniqueName: \"kubernetes.io/projected/5ecf7f83-bf59-427b-af68-07b70f1d7385-kube-api-access-mvb69\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.099055 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.099036    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf7f83-bf59-427b-af68-07b70f1d7385-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.154970 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.154939    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-97c87c-serving-cert\""
Nov 01 14:59:38.157250 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.157230    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"]
Nov 01 14:59:38.200301 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.200263    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.200514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.200320    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvb69\" (UniqueName: \"kubernetes.io/projected/5ecf7f83-bf59-427b-af68-07b70f1d7385-kube-api-access-mvb69\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.200514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.200426    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf7f83-bf59-427b-af68-07b70f1d7385-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.200514 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:38.200441    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-serving-cert: secret "maas-api-e2e-shared-a-w1-97c87c-serving-cert" not found
Nov 01 14:59:38.200514 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.200472    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.200747 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:38.200526    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls podName:5ecf7f83-bf59-427b-af68-07b70f1d7385 nodeName:}" failed. No retries permitted until 2026-11-01 14:59:38.700504147 +0000 UTC m=+2518.865989443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls") pod "maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f" (UID: "5ecf7f83-bf59-427b-af68-07b70f1d7385") : secret "maas-api-e2e-shared-a-w1-97c87c-serving-cert" not found
Nov 01 14:59:38.201468 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.201444    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf7f83-bf59-427b-af68-07b70f1d7385-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.203118 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.203094    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.260293 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.260255    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvb69\" (UniqueName: \"kubernetes.io/projected/5ecf7f83-bf59-427b-af68-07b70f1d7385-kube-api-access-mvb69\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.726455 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.726415    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.729076 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.729049    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:38.963515 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:38.963483    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:39.333685 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:59:39.333657    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecf7f83_bf59_427b_af68_07b70f1d7385.slice/crio-08667badfc4567079a6f6ee7238afbc076e14121dde040195f7d3ece7d1c720c WatchSource:0}: Error finding container 08667badfc4567079a6f6ee7238afbc076e14121dde040195f7d3ece7d1c720c: Status 404 returned error can't find the container with id 08667badfc4567079a6f6ee7238afbc076e14121dde040195f7d3ece7d1c720c
Nov 01 14:59:39.336602 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:39.336578    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"]
Nov 01 14:59:39.926683 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:39.926644    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f" event={"ID":"5ecf7f83-bf59-427b-af68-07b70f1d7385","Type":"ContainerStarted","Data":"08667badfc4567079a6f6ee7238afbc076e14121dde040195f7d3ece7d1c720c"}
Nov 01 14:59:40.933910 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:40.933867    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f" event={"ID":"5ecf7f83-bf59-427b-af68-07b70f1d7385","Type":"ContainerStarted","Data":"e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12"}
Nov 01 14:59:40.934342 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:40.933972    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:41.051793 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:41.051736    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f" podStartSLOduration=2.139935891 podStartE2EDuration="3.0517228s" totalImagesPullingTime="911.786909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:41.051216681 +0000 UTC m=+2521.216701979" watchObservedRunningTime="2026-11-01 14:59:41.0517228 +0000 UTC m=+2521.217208166"
Nov 01 14:59:45.067259 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:45.067223    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 14:59:46.944831 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:46.944801    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 14:59:52.036821 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.036746    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"]
Nov 01 14:59:52.042487 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.042463    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.097837 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.097806    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"]
Nov 01 14:59:52.114498 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.114460    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqkn\" (UniqueName: \"kubernetes.io/projected/2e7b704b-d872-4d1c-bca8-d8347330da3b-kube-api-access-xtqkn\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.114720 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.114699    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.114806 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.114791    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7b704b-d872-4d1c-bca8-d8347330da3b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.114872 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.114860    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.145776 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.145744    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-d06633-serving-cert\""
Nov 01 14:59:52.216210 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.216180    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.216402 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.216238    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7b704b-d872-4d1c-bca8-d8347330da3b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.216402 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.216277    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.216402 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.216308    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqkn\" (UniqueName: \"kubernetes.io/projected/2e7b704b-d872-4d1c-bca8-d8347330da3b-kube-api-access-xtqkn\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.216402 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:52.216344    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-serving-cert: secret "maas-api-e2e-shared-a-w4-d06633-serving-cert" not found
Nov 01 14:59:52.216645 ip-10-0-135-142 kubenswrapper[5423]: E1101 14:59:52.216437    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls podName:2e7b704b-d872-4d1c-bca8-d8347330da3b nodeName:}" failed. No retries permitted until 2026-11-01 14:59:52.716415696 +0000 UTC m=+2532.881900993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls") pod "maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8" (UID: "2e7b704b-d872-4d1c-bca8-d8347330da3b") : secret "maas-api-e2e-shared-a-w4-d06633-serving-cert" not found
Nov 01 14:59:52.216995 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.216970    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7b704b-d872-4d1c-bca8-d8347330da3b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.218883 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.218854    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.307454 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.306131    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqkn\" (UniqueName: \"kubernetes.io/projected/2e7b704b-d872-4d1c-bca8-d8347330da3b-kube-api-access-xtqkn\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.722210 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.722127    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.724767 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.724739    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:52.957980 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:52.957938    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:53.338017 ip-10-0-135-142 kubenswrapper[5423]: W1101 14:59:53.337980    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7b704b_d872_4d1c_bca8_d8347330da3b.slice/crio-2a1253921fbc3326dedf64429bb49ff52a7d95da9efaf9690f4516d7c9d92c3b WatchSource:0}: Error finding container 2a1253921fbc3326dedf64429bb49ff52a7d95da9efaf9690f4516d7c9d92c3b: Status 404 returned error can't find the container with id 2a1253921fbc3326dedf64429bb49ff52a7d95da9efaf9690f4516d7c9d92c3b
Nov 01 14:59:53.340946 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:53.340920    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"]
Nov 01 14:59:54.020647 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:54.020604    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8" event={"ID":"2e7b704b-d872-4d1c-bca8-d8347330da3b","Type":"ContainerStarted","Data":"2a1253921fbc3326dedf64429bb49ff52a7d95da9efaf9690f4516d7c9d92c3b"}
Nov 01 14:59:54.149545 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:54.149499    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:59:54.149745 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:54.149576    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:59:55.029277 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:55.029232    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8" event={"ID":"2e7b704b-d872-4d1c-bca8-d8347330da3b","Type":"ContainerStarted","Data":"d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b"}
Nov 01 14:59:55.029759 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:55.029313    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 14:59:55.147732 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:55.147675    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8" podStartSLOduration=3.249049805 podStartE2EDuration="4.147661257s" totalImagesPullingTime="898.611452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:55.147000118 +0000 UTC m=+2535.312485418" watchObservedRunningTime="2026-11-01 14:59:55.147661257 +0000 UTC m=+2535.313146554"
Nov 01 14:59:56.036270 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:56.036234    5423 generic.go:356] "Generic (PLEG): container finished" podID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerID="609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32" exitCode=0
Nov 01 14:59:56.036715 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:56.036308    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" event={"ID":"bbb5f2ae-d498-4cde-95be-e7dd1339c347","Type":"ContainerDied","Data":"609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32"}
Nov 01 14:59:56.036801 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:56.036788    5423 scope.go:122] "RemoveContainer" containerID="609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32"
Nov 01 14:59:56.038042 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:56.038017    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:59:56.038099 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:56.038080    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:59:56.038137 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:56.038111    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 14:59:57.060712 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:57.060678    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" event={"ID":"bbb5f2ae-d498-4cde-95be-e7dd1339c347","Type":"ContainerStarted","Data":"e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c"}
Nov 01 14:59:57.071829 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:57.071798    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 14:59:57.072179 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:57.072154    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 14:59:57.072328 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:57.072305    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 14:59:58.071660 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:58.071626    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 14:59:58.072062 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:58.071690    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 14:59:59.070942 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:59.070893    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 14:59:59.071165 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:59.070979    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 14:59:59.880928 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:59.880892    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"]
Nov 01 14:59:59.885735 ip-10-0-135-142 kubenswrapper[5423]: I1101 14:59:59.885716    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.043408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.041808    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-6fcb7b-openshift-default-dockercfg-shm7r\""
Nov 01 15:00:00.092411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.091848    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 15:00:00.092411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.091908    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 15:00:00.176125 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.176038    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"]
Nov 01 15:00:00.180261 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.180235    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istiod-ca-cert\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.180490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.180474    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-certs\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.180653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.180640    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqmb\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-kube-api-access-rzqmb\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.180748 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.180736    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-data\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.180867 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.180853    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-envoy\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.180941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.180929    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-socket\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.181076 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.181065    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-token\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.181214 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.181202    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-credential-socket\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.181318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.181307    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-podinfo\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.230316 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.230290    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"]
Nov 01 15:00:00.235548 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.235521    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:00.273753 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.273719    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"]
Nov 01 15:00:00.282714 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.282679    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqmb\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-kube-api-access-rzqmb\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.282914 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.282893    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-data\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.283420 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.283378    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-envoy\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.283577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.283561    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-socket\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.283779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.283764    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-token\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.283905 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.283889    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-credential-socket\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.284036 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.284019    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz9n7\" (UniqueName: \"kubernetes.io/projected/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-kube-api-access-hz9n7\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww\" (UID: \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:00.284728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.283886    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-socket\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.284728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.284133    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-podinfo\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.284728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.284226    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istiod-ca-cert\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.284728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.284258    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-certs\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.284728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.284312    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww\" (UID: \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:00.284728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.283379    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-data\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.284728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.284684    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-credential-socket\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.285167 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.284853    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-certs\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.285358 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.285331    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istiod-ca-cert\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.286991 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.286967    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-envoy\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.287176 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.287157    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-podinfo\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.366107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.366064    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 15:00:00.378375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.377955    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-token\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.379639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.379592    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqmb\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-kube-api-access-rzqmb\") pod \"e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") " pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.385229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.385195    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz9n7\" (UniqueName: \"kubernetes.io/projected/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-kube-api-access-hz9n7\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww\" (UID: \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:00.385370 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.385313    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww\" (UID: \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:00.385923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.385903    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww\" (UID: \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:00.445375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.443911    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz9n7\" (UniqueName: \"kubernetes.io/projected/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-kube-api-access-hz9n7\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww\" (UID: \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:00.517410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.507605    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:00.553810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.553772    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:00.935278 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:00:00.935249    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78e9ac0_aaa0_4a41_8122_f7dc9bb49154.slice/crio-39bb4c22ae133488f7d30b1aaebe571d2abbdd4a4cab39c924764aedad45e9fa WatchSource:0}: Error finding container 39bb4c22ae133488f7d30b1aaebe571d2abbdd4a4cab39c924764aedad45e9fa: Status 404 returned error can't find the container with id 39bb4c22ae133488f7d30b1aaebe571d2abbdd4a4cab39c924764aedad45e9fa
Nov 01 15:00:00.937622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.937597    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"]
Nov 01 15:00:00.937926 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.937894    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:00.938015 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.937957    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:00.938015 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.937982    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:00.986810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:00.986782    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"]
Nov 01 15:00:01.043715 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:01.043381    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 15:00:01.072087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:01.072055    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 15:00:01.072212 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:01.072104    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 15:00:01.185986 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:01.185903    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" event={"ID":"770a1143-75f6-48e9-a0e3-59ca4ab94cbd","Type":"ContainerStarted","Data":"1410011f6c6892fecff9756b6889acd16272522eed1dcbe914d9cf655f9c3e00"}
Nov 01 15:00:01.187661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:01.187623    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" event={"ID":"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154","Type":"ContainerStarted","Data":"e7a76aa55d023232c4068a3d9b883256ce26e23e3b7c26ca2ccc1348b6e0e882"}
Nov 01 15:00:01.187661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:01.187662    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" event={"ID":"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154","Type":"ContainerStarted","Data":"39bb4c22ae133488f7d30b1aaebe571d2abbdd4a4cab39c924764aedad45e9fa"}
Nov 01 15:00:01.317613 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:01.317544    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" podStartSLOduration=2.317522939 podStartE2EDuration="2.317522939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:01.315881455 +0000 UTC m=+2541.481366754" watchObservedRunningTime="2026-11-01 15:00:01.317522939 +0000 UTC m=+2541.483008238"
Nov 01 15:00:01.522582 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:01.522152    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:02.071309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:02.071273    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 15:00:02.071693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:02.071338    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 15:00:02.195358 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:02.195317    5423 generic.go:356] "Generic (PLEG): container finished" podID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerID="e9c5d884d788c1cca6c2235fb27859464fc4ff01c34825563ffd08e412acfe83" exitCode=60
Nov 01 15:00:02.195518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:02.195406    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" event={"ID":"770a1143-75f6-48e9-a0e3-59ca4ab94cbd","Type":"ContainerDied","Data":"e9c5d884d788c1cca6c2235fb27859464fc4ff01c34825563ffd08e412acfe83"}
Nov 01 15:00:02.195731 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:02.195713    5423 scope.go:122] "RemoveContainer" containerID="e9c5d884d788c1cca6c2235fb27859464fc4ff01c34825563ffd08e412acfe83"
Nov 01 15:00:02.520963 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:02.520940    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:03.075909 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.075877    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 15:00:03.076371 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.076176    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 15:00:03.076974 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.076959    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 15:00:03.207244 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.207159    5423 generic.go:356] "Generic (PLEG): container finished" podID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerID="0daab45e2f4b82ab81cf145eb557a57cf1a8624d56cd4c599fc3a32f35d7d3bf" exitCode=60
Nov 01 15:00:03.207429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.207239    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" event={"ID":"770a1143-75f6-48e9-a0e3-59ca4ab94cbd","Type":"ContainerDied","Data":"0daab45e2f4b82ab81cf145eb557a57cf1a8624d56cd4c599fc3a32f35d7d3bf"}
Nov 01 15:00:03.207429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.207294    5423 scope.go:122] "RemoveContainer" containerID="e9c5d884d788c1cca6c2235fb27859464fc4ff01c34825563ffd08e412acfe83"
Nov 01 15:00:03.207635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.207620    5423 scope.go:122] "RemoveContainer" containerID="0daab45e2f4b82ab81cf145eb557a57cf1a8624d56cd4c599fc3a32f35d7d3bf"
Nov 01 15:00:03.207775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.207751    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:03.207963 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:00:03.207941    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww_odh-ai-gateway-infra(770a1143-75f6-48e9-a0e3-59ca4ab94cbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd"
Nov 01 15:00:03.208727 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:03.208609    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:00:04.215066 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:04.215038    5423 scope.go:122] "RemoveContainer" containerID="0daab45e2f4b82ab81cf145eb557a57cf1a8624d56cd4c599fc3a32f35d7d3bf"
Nov 01 15:00:04.215478 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:00:04.215328    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww_odh-ai-gateway-infra(770a1143-75f6-48e9-a0e3-59ca4ab94cbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd"
Nov 01 15:00:05.155837 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:00:05.155799    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:00:09.045749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.045708    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"]
Nov 01 15:00:09.049859 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.049840    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:09.106935 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.106890    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"]
Nov 01 15:00:09.145410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.140263    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496ebcf6-2c66-409a-8d43-561bf1e1118d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt\" (UID: \"496ebcf6-2c66-409a-8d43-561bf1e1118d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:09.145410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.140458    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4pn\" (UniqueName: \"kubernetes.io/projected/496ebcf6-2c66-409a-8d43-561bf1e1118d-kube-api-access-9k4pn\") pod \"payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt\" (UID: \"496ebcf6-2c66-409a-8d43-561bf1e1118d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:09.175422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.171864    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-6fcb7b-dockercfg-w5kbr\""
Nov 01 15:00:09.175422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.171945    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6fcb7b\""
Nov 01 15:00:09.246117 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.246037    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496ebcf6-2c66-409a-8d43-561bf1e1118d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt\" (UID: \"496ebcf6-2c66-409a-8d43-561bf1e1118d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:09.246667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.246634    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4pn\" (UniqueName: \"kubernetes.io/projected/496ebcf6-2c66-409a-8d43-561bf1e1118d-kube-api-access-9k4pn\") pod \"payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt\" (UID: \"496ebcf6-2c66-409a-8d43-561bf1e1118d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:09.247158 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.247102    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496ebcf6-2c66-409a-8d43-561bf1e1118d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt\" (UID: \"496ebcf6-2c66-409a-8d43-561bf1e1118d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:09.247648 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.247627    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"]
Nov 01 15:00:09.252324 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.252308    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:09.275366 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.275341    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"]
Nov 01 15:00:09.304774 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.304715    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4pn\" (UniqueName: \"kubernetes.io/projected/496ebcf6-2c66-409a-8d43-561bf1e1118d-kube-api-access-9k4pn\") pod \"payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt\" (UID: \"496ebcf6-2c66-409a-8d43-561bf1e1118d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:09.348352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.348319    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59pcs\" (UniqueName: \"kubernetes.io/projected/3ca03e2f-f398-4a2c-b1ce-f86620269544-kube-api-access-59pcs\") pod \"payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2\" (UID: \"3ca03e2f-f398-4a2c-b1ce-f86620269544\") " pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:09.348510 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.348447    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca03e2f-f398-4a2c-b1ce-f86620269544-plugins-config-volume\") pod \"payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2\" (UID: \"3ca03e2f-f398-4a2c-b1ce-f86620269544\") " pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:09.363012 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.362987    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:09.450159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.450127    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pcs\" (UniqueName: \"kubernetes.io/projected/3ca03e2f-f398-4a2c-b1ce-f86620269544-kube-api-access-59pcs\") pod \"payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2\" (UID: \"3ca03e2f-f398-4a2c-b1ce-f86620269544\") " pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:09.450306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.450232    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca03e2f-f398-4a2c-b1ce-f86620269544-plugins-config-volume\") pod \"payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2\" (UID: \"3ca03e2f-f398-4a2c-b1ce-f86620269544\") " pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:09.451045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.451019    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca03e2f-f398-4a2c-b1ce-f86620269544-plugins-config-volume\") pod \"payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2\" (UID: \"3ca03e2f-f398-4a2c-b1ce-f86620269544\") " pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:09.508930 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.508898    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pcs\" (UniqueName: \"kubernetes.io/projected/3ca03e2f-f398-4a2c-b1ce-f86620269544-kube-api-access-59pcs\") pod \"payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2\" (UID: \"3ca03e2f-f398-4a2c-b1ce-f86620269544\") " pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:09.574028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.573993    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:09.729902 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:00:09.729872    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496ebcf6_2c66_409a_8d43_561bf1e1118d.slice/crio-0400c9c1321293f6830730a12cefa04408a300d5c0252b29e74799ee44f1a6f9 WatchSource:0}: Error finding container 0400c9c1321293f6830730a12cefa04408a300d5c0252b29e74799ee44f1a6f9: Status 404 returned error can't find the container with id 0400c9c1321293f6830730a12cefa04408a300d5c0252b29e74799ee44f1a6f9
Nov 01 15:00:09.733366 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.733340    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"]
Nov 01 15:00:09.949181 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:00:09.949147    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca03e2f_f398_4a2c_b1ce_f86620269544.slice/crio-721e79688a0885058d645ee621d5219138faaf12b527d86cfdb1e908580e55c9 WatchSource:0}: Error finding container 721e79688a0885058d645ee621d5219138faaf12b527d86cfdb1e908580e55c9: Status 404 returned error can't find the container with id 721e79688a0885058d645ee621d5219138faaf12b527d86cfdb1e908580e55c9
Nov 01 15:00:09.951989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:09.951965    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"]
Nov 01 15:00:10.266355 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:10.266266    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2" event={"ID":"3ca03e2f-f398-4a2c-b1ce-f86620269544","Type":"ContainerStarted","Data":"1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa"}
Nov 01 15:00:10.266355 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:10.266327    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2" event={"ID":"3ca03e2f-f398-4a2c-b1ce-f86620269544","Type":"ContainerStarted","Data":"721e79688a0885058d645ee621d5219138faaf12b527d86cfdb1e908580e55c9"}
Nov 01 15:00:10.266842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:10.266355    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:10.268066 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:10.268031    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt" event={"ID":"496ebcf6-2c66-409a-8d43-561bf1e1118d","Type":"ContainerStarted","Data":"c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50"}
Nov 01 15:00:10.268180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:10.268090    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:10.268180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:10.268109    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt" event={"ID":"496ebcf6-2c66-409a-8d43-561bf1e1118d","Type":"ContainerStarted","Data":"0400c9c1321293f6830730a12cefa04408a300d5c0252b29e74799ee44f1a6f9"}
Nov 01 15:00:10.404407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:10.397363    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2" podStartSLOduration=1.397342001 podStartE2EDuration="1.397342001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:10.391645789 +0000 UTC m=+2550.557131090" watchObservedRunningTime="2026-11-01 15:00:10.397342001 +0000 UTC m=+2550.562827301"
Nov 01 15:00:10.456871 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:10.456832    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:00:16.953986 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:16.953955    5423 scope.go:122] "RemoveContainer" containerID="0daab45e2f4b82ab81cf145eb557a57cf1a8624d56cd4c599fc3a32f35d7d3bf"
Nov 01 15:00:17.318518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:17.318491    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" event={"ID":"770a1143-75f6-48e9-a0e3-59ca4ab94cbd","Type":"ContainerStarted","Data":"0632537be7eda9b65f87b345bdd868539eac2309a5d07c27b4e529e8a2f03684"}
Nov 01 15:00:17.443667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:17.443613    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" podStartSLOduration=17.443599225 podStartE2EDuration="17.443599225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:17.442157109 +0000 UTC m=+2557.607642404" watchObservedRunningTime="2026-11-01 15:00:17.443599225 +0000 UTC m=+2557.609084522"
Nov 01 15:00:17.444487 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:17.444458    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt" podStartSLOduration=8.444450889 podStartE2EDuration="8.444450889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:10.530233753 +0000 UTC m=+2550.695719053" watchObservedRunningTime="2026-11-01 15:00:17.444450889 +0000 UTC m=+2557.609936231"
Nov 01 15:00:18.328032 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:18.327995    5423 generic.go:356] "Generic (PLEG): container finished" podID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerID="0632537be7eda9b65f87b345bdd868539eac2309a5d07c27b4e529e8a2f03684" exitCode=60
Nov 01 15:00:18.328469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:18.328059    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" event={"ID":"770a1143-75f6-48e9-a0e3-59ca4ab94cbd","Type":"ContainerDied","Data":"0632537be7eda9b65f87b345bdd868539eac2309a5d07c27b4e529e8a2f03684"}
Nov 01 15:00:18.328469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:18.328119    5423 scope.go:122] "RemoveContainer" containerID="0daab45e2f4b82ab81cf145eb557a57cf1a8624d56cd4c599fc3a32f35d7d3bf"
Nov 01 15:00:18.328615 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:18.328490    5423 scope.go:122] "RemoveContainer" containerID="0632537be7eda9b65f87b345bdd868539eac2309a5d07c27b4e529e8a2f03684"
Nov 01 15:00:18.328866 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:00:18.328841    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww_odh-ai-gateway-infra(770a1143-75f6-48e9-a0e3-59ca4ab94cbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd"
Nov 01 15:00:18.464463 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:18.464421    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"]
Nov 01 15:00:19.491945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:19.491918    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:19.603143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:19.603069    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-maas-api-service-ca\") pod \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\" (UID: \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\") "
Nov 01 15:00:19.603303 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:19.603165    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-kube-api-access-hz9n7\" (UniqueName: \"kubernetes.io/projected/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-kube-api-access-hz9n7\") pod \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\" (UID: \"770a1143-75f6-48e9-a0e3-59ca4ab94cbd\") "
Nov 01 15:00:19.603457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:19.603419    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-maas-api-service-ca" pod "770a1143-75f6-48e9-a0e3-59ca4ab94cbd" (UID: "770a1143-75f6-48e9-a0e3-59ca4ab94cbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:00:19.605315 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:19.605287    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-kube-api-access-hz9n7" pod "770a1143-75f6-48e9-a0e3-59ca4ab94cbd" (UID: "770a1143-75f6-48e9-a0e3-59ca4ab94cbd"). InnerVolumeSpecName "kube-api-access-hz9n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:00:19.704322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:19.704297    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:00:19.704322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:19.704323    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz9n7\" (UniqueName: \"kubernetes.io/projected/770a1143-75f6-48e9-a0e3-59ca4ab94cbd-kube-api-access-hz9n7\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:00:20.345746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:20.345702    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww" event={"ID":"770a1143-75f6-48e9-a0e3-59ca4ab94cbd","Type":"ContainerDied","Data":"1410011f6c6892fecff9756b6889acd16272522eed1dcbe914d9cf655f9c3e00"}
Nov 01 15:00:20.345929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:20.345781    5423 scope.go:122] "RemoveContainer" containerID="0632537be7eda9b65f87b345bdd868539eac2309a5d07c27b4e529e8a2f03684"
Nov 01 15:00:20.345929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:20.345906    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"
Nov 01 15:00:20.516634 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:20.516597    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"]
Nov 01 15:00:20.519941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:20.519920    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892420-nt8ww"]
Nov 01 15:00:21.281316 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:21.281287    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:00:21.281492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:21.281363    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:00:21.961191 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:21.961159    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" path="/var/lib/kubelet/pods/770a1143-75f6-48e9-a0e3-59ca4ab94cbd/volumes"
Nov 01 15:00:23.849567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.849534    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"]
Nov 01 15:00:23.850092 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.850078    5423 cpu_manager.go:414] "containerMap: removing container" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:23.850138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.850093    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:23.850138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.850107    5423 cpu_manager.go:414] "containerMap: removing container" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:23.850138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.850112    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:23.850233 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.850229    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:23.850267 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.850241    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:23.850267 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.850250    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:23.854858 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.854842    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.908426 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.908375    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-d06633-openshift-default-dockercfg-lk8zc\""
Nov 01 15:00:23.956632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.956598    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-credential-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.956815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.956756    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-istio-token\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.956815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.956807    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xsb\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-kube-api-access-q8xsb\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.956934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.956862    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512a93a5-2762-44db-9a84-156bea8770f4-istiod-ca-cert\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.956994 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.956953    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-data\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.957051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.956992    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-certs\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.957112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.957069    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.957112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.957094    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-envoy\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.957211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.957166    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512a93a5-2762-44db-9a84-156bea8770f4-istio-podinfo\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:23.963717 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:23.963687    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"]
Nov 01 15:00:24.058250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058212    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xsb\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-kube-api-access-q8xsb\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058298    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512a93a5-2762-44db-9a84-156bea8770f4-istiod-ca-cert\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058355    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-data\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058406    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-certs\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058446    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058483    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-envoy\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058545    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512a93a5-2762-44db-9a84-156bea8770f4-istio-podinfo\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058622    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-credential-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058735    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-istio-token\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.058890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.058881    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-data\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.059347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.059154    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-credential-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.059347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.059302    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-certs\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.059530 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.059343    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.059839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.059817    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512a93a5-2762-44db-9a84-156bea8770f4-istiod-ca-cert\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.061194 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.061165    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-envoy\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.061509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.061491    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512a93a5-2762-44db-9a84-156bea8770f4-istio-podinfo\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.121030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.120884    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xsb\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-kube-api-access-q8xsb\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.128870 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.128837    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-istio-token\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.168324 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.168294    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:24.595143 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:00:24.595114    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512a93a5_2762_44db_9a84_156bea8770f4.slice/crio-ac34559bec93ffaecd3f2dc28bbc968df62a849ac703d2d9f0e2ebc2dbdc8ae6 WatchSource:0}: Error finding container ac34559bec93ffaecd3f2dc28bbc968df62a849ac703d2d9f0e2ebc2dbdc8ae6: Status 404 returned error can't find the container with id ac34559bec93ffaecd3f2dc28bbc968df62a849ac703d2d9f0e2ebc2dbdc8ae6
Nov 01 15:00:24.597862 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.597824    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:24.597964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.597895    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:24.597964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.597928    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:24.598325 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:24.598302    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"]
Nov 01 15:00:25.381488 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:25.381443    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" event={"ID":"512a93a5-2762-44db-9a84-156bea8770f4","Type":"ContainerStarted","Data":"4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974"}
Nov 01 15:00:25.381929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:25.381536    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" event={"ID":"512a93a5-2762-44db-9a84-156bea8770f4","Type":"ContainerStarted","Data":"ac34559bec93ffaecd3f2dc28bbc968df62a849ac703d2d9f0e2ebc2dbdc8ae6"}
Nov 01 15:00:25.553180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:25.553123    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" podStartSLOduration=2.553108802 podStartE2EDuration="2.553108802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:25.551109912 +0000 UTC m=+2565.716595209" watchObservedRunningTime="2026-11-01 15:00:25.553108802 +0000 UTC m=+2565.718594158"
Nov 01 15:00:26.169174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:26.169144    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:26.170884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:26.170857    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 15:00:26.171025 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:26.170908    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 15:00:27.172839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:27.172800    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 15:00:27.173517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:27.173481    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 15:00:28.168981 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:28.168945    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 15:00:28.169161 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:28.169007    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 15:00:29.168958 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:29.168922    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 15:00:29.169372 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:29.168974    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 15:00:30.171325 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:30.171207    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 01 15:00:30.171325 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:30.171278    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 01 15:00:31.173113 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:31.173083    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:31.173538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:31.173368    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:31.174028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:31.174010    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:00:35.193614 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:00:35.192978    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:00:35.237915 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.237876    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"]
Nov 01 15:00:35.238560 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.238540    5423 cpu_manager.go:414] "containerMap: removing container" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:35.238685 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.238561    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770a1143-75f6-48e9-a0e3-59ca4ab94cbd" containerName="cleanup"
Nov 01 15:00:35.243570 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.243548    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.297166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.297136    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-52fffa-openshift-default-dockercfg-dkt7p\""
Nov 01 15:00:35.352429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.352379    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"]
Nov 01 15:00:35.365833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.365804    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-certs\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.366014 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.365934    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-istio-token\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.366014 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.365971    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0966ad2c-b364-4404-9fdc-479744097176-istio-podinfo\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.366145 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.366044    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-data\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.366145 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.366127    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-socket\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.366264 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.366177    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0966ad2c-b364-4404-9fdc-479744097176-istiod-ca-cert\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.366264 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.366226    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-credential-socket\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.366374 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.366264    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwd2\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-kube-api-access-qrwd2\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.366374 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.366311    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-envoy\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.467597 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467518    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-istio-token\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.467597 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467562    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0966ad2c-b364-4404-9fdc-479744097176-istio-podinfo\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.467597 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467591    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-data\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.467848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467656    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-socket\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.467848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467694    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0966ad2c-b364-4404-9fdc-479744097176-istiod-ca-cert\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.467848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467753    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-credential-socket\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.467848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467792    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwd2\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-kube-api-access-qrwd2\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.467848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467825    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-envoy\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.468124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.467864    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-certs\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.468124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.468114    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-data\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.468246 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.468198    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-socket\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.468309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.468268    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-certs\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.468485 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.468457    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-credential-socket\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.468594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.468527    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0966ad2c-b364-4404-9fdc-479744097176-istiod-ca-cert\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.470171 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.470148    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-envoy\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.470266 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.470240    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0966ad2c-b364-4404-9fdc-479744097176-istio-podinfo\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.537409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.537214    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwd2\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-kube-api-access-qrwd2\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.569493 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.565914    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-istio-token\") pod \"e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") " pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:35.569493 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:35.566170    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:36.036131 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:36.036095    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:36.036225 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:36.036168    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:36.036225 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:36.036194    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:00:36.038745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:36.038722    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"]
Nov 01 15:00:36.688117 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:36.688073    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" event={"ID":"0966ad2c-b364-4404-9fdc-479744097176","Type":"ContainerStarted","Data":"42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f"}
Nov 01 15:00:36.688539 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:36.688132    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" event={"ID":"0966ad2c-b364-4404-9fdc-479744097176","Type":"ContainerStarted","Data":"ba87847cee8edd54e267bb0d55990b8490a0925f60561741a6ce18701f40a071"}
Nov 01 15:00:36.814306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:36.814242    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" podStartSLOduration=1.81422452 podStartE2EDuration="1.81422452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:36.81168261 +0000 UTC m=+2576.977167909" watchObservedRunningTime="2026-11-01 15:00:36.81422452 +0000 UTC m=+2576.979709821"
Nov 01 15:00:37.568329 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:37.568284    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:37.573811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:37.573784    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:37.694547 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:37.694510    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:37.695497 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:37.695478    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:00:43.001098 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.001066    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"]
Nov 01 15:00:43.006552 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.006528    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.088875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.088840    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhcnn\" (UniqueName: \"kubernetes.io/projected/f364e0c0-495f-4818-b6e5-e336fbab1ab6-kube-api-access-jhcnn\") pod \"payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm\" (UID: \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.095989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.095954    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"]
Nov 01 15:00:43.096517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.096490    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f364e0c0-495f-4818-b6e5-e336fbab1ab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm\" (UID: \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.112711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.112666    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-52fffa\""
Nov 01 15:00:43.114641 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.114619    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-52fffa-dockercfg-lt75p\""
Nov 01 15:00:43.215447 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.215415    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"]
Nov 01 15:00:43.217542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.217434    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhcnn\" (UniqueName: \"kubernetes.io/projected/f364e0c0-495f-4818-b6e5-e336fbab1ab6-kube-api-access-jhcnn\") pod \"payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm\" (UID: \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.225535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.222769    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f364e0c0-495f-4818-b6e5-e336fbab1ab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm\" (UID: \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.225535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.223926    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f364e0c0-495f-4818-b6e5-e336fbab1ab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm\" (UID: \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.230482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.227550    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:43.250974 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.250947    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"]
Nov 01 15:00:43.284170 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.284097    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhcnn\" (UniqueName: \"kubernetes.io/projected/f364e0c0-495f-4818-b6e5-e336fbab1ab6-kube-api-access-jhcnn\") pod \"payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm\" (UID: \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.319628 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.319595    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.323743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.323711    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bgfq\" (UniqueName: \"kubernetes.io/projected/fabf9cde-981f-4801-be3e-7ba67fa35f77-kube-api-access-2bgfq\") pod \"payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2\" (UID: \"fabf9cde-981f-4801-be3e-7ba67fa35f77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:43.323958 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.323941    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fabf9cde-981f-4801-be3e-7ba67fa35f77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2\" (UID: \"fabf9cde-981f-4801-be3e-7ba67fa35f77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:43.425648 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.425615    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fabf9cde-981f-4801-be3e-7ba67fa35f77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2\" (UID: \"fabf9cde-981f-4801-be3e-7ba67fa35f77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:43.425884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.425728    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bgfq\" (UniqueName: \"kubernetes.io/projected/fabf9cde-981f-4801-be3e-7ba67fa35f77-kube-api-access-2bgfq\") pod \"payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2\" (UID: \"fabf9cde-981f-4801-be3e-7ba67fa35f77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:43.426658 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.426624    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fabf9cde-981f-4801-be3e-7ba67fa35f77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2\" (UID: \"fabf9cde-981f-4801-be3e-7ba67fa35f77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:43.486425 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.486376    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bgfq\" (UniqueName: \"kubernetes.io/projected/fabf9cde-981f-4801-be3e-7ba67fa35f77-kube-api-access-2bgfq\") pod \"payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2\" (UID: \"fabf9cde-981f-4801-be3e-7ba67fa35f77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:43.554232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.554163    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:43.694023 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:00:43.693977    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf364e0c0_495f_4818_b6e5_e336fbab1ab6.slice/crio-1e32f71e74f83956ec381ab828b806658eab00e9c0924714621314ecd2165784 WatchSource:0}: Error finding container 1e32f71e74f83956ec381ab828b806658eab00e9c0924714621314ecd2165784: Status 404 returned error can't find the container with id 1e32f71e74f83956ec381ab828b806658eab00e9c0924714621314ecd2165784
Nov 01 15:00:43.697101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.697077    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"]
Nov 01 15:00:43.771577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.771550    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm" event={"ID":"f364e0c0-495f-4818-b6e5-e336fbab1ab6","Type":"ContainerStarted","Data":"f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936"}
Nov 01 15:00:43.771700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.771602    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm" event={"ID":"f364e0c0-495f-4818-b6e5-e336fbab1ab6","Type":"ContainerStarted","Data":"1e32f71e74f83956ec381ab828b806658eab00e9c0924714621314ecd2165784"}
Nov 01 15:00:43.771700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.771623    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:43.891323 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.891272    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm" podStartSLOduration=1.891258219 podStartE2EDuration="1.891258219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:43.888714595 +0000 UTC m=+2584.054199895" watchObservedRunningTime="2026-11-01 15:00:43.891258219 +0000 UTC m=+2584.056743517"
Nov 01 15:00:43.925904 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:00:43.925872    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabf9cde_981f_4801_be3e_7ba67fa35f77.slice/crio-f3ae25c4f431914398fa01004a480ae336e1a17bd7ddb886e8ef28e15c14ce6b WatchSource:0}: Error finding container f3ae25c4f431914398fa01004a480ae336e1a17bd7ddb886e8ef28e15c14ce6b: Status 404 returned error can't find the container with id f3ae25c4f431914398fa01004a480ae336e1a17bd7ddb886e8ef28e15c14ce6b
Nov 01 15:00:43.927570 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:43.927532    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"]
Nov 01 15:00:44.801489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:44.801445    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2" event={"ID":"fabf9cde-981f-4801-be3e-7ba67fa35f77","Type":"ContainerStarted","Data":"adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53"}
Nov 01 15:00:44.802159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:44.802131    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2" event={"ID":"fabf9cde-981f-4801-be3e-7ba67fa35f77","Type":"ContainerStarted","Data":"f3ae25c4f431914398fa01004a480ae336e1a17bd7ddb886e8ef28e15c14ce6b"}
Nov 01 15:00:44.802329 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:44.802314    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:44.932028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:44.931952    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2" podStartSLOduration=1.9319343820000001 podStartE2EDuration="1.931934382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:44.928402734 +0000 UTC m=+2585.093888026" watchObservedRunningTime="2026-11-01 15:00:44.931934382 +0000 UTC m=+2585.097419681"
Nov 01 15:00:54.807606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:54.807572    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:00:56.854408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:56.850569    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:00:58.717885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:58.717852    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"]
Nov 01 15:00:58.723054 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:58.723033    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:00:58.901168 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:58.901132    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b3f06c-dockercfg-kbb9q\""
Nov 01 15:00:58.901477 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:58.901458    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b3f06c\""
Nov 01 15:00:58.909293 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:58.909220    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m\" (UID: \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:00:58.910018 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:58.909989    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mw95\" (UniqueName: \"kubernetes.io/projected/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-kube-api-access-5mw95\") pod \"payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m\" (UID: \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:00:58.914852 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:58.914817    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"]
Nov 01 15:00:59.011712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:59.011626    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m\" (UID: \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:00:59.011712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:59.011691    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mw95\" (UniqueName: \"kubernetes.io/projected/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-kube-api-access-5mw95\") pod \"payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m\" (UID: \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:00:59.012489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:59.012464    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m\" (UID: \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:00:59.071210 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:59.071181    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mw95\" (UniqueName: \"kubernetes.io/projected/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-kube-api-access-5mw95\") pod \"payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m\" (UID: \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:00:59.338129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:59.338097    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:00:59.709116 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:00:59.709075    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2ae3bd_5205_41e3_a0b1_8c2fc1b4dfe3.slice/crio-fd716e0cfe1154626f7083ab387182e30374e549f715a3dc285dcaf967bd70e6 WatchSource:0}: Error finding container fd716e0cfe1154626f7083ab387182e30374e549f715a3dc285dcaf967bd70e6: Status 404 returned error can't find the container with id fd716e0cfe1154626f7083ab387182e30374e549f715a3dc285dcaf967bd70e6
Nov 01 15:00:59.712699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:00:59.712654    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"]
Nov 01 15:01:00.012506 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:00.012418    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m" event={"ID":"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3","Type":"ContainerStarted","Data":"579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597"}
Nov 01 15:01:00.012506 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:00.012481    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m" event={"ID":"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3","Type":"ContainerStarted","Data":"fd716e0cfe1154626f7083ab387182e30374e549f715a3dc285dcaf967bd70e6"}
Nov 01 15:01:00.012506 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:00.012504    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:01:00.132013 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:00.131955    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m" podStartSLOduration=2.131942197 podStartE2EDuration="2.131942197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:01:00.130504012 +0000 UTC m=+2600.295989310" watchObservedRunningTime="2026-11-01 15:01:00.131942197 +0000 UTC m=+2600.297427495"
Nov 01 15:01:05.229040 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:01:05.228961    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:01:05.405841 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.405804    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"]
Nov 01 15:01:05.410253 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.410229    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:05.467963 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.467932    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"]
Nov 01 15:01:05.508488 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.503345    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-kube-api-access-m9rqq\") pod \"payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f\" (UID: \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:05.508488 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.506670    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f\" (UID: \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:05.521451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.516619    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-8d6ea2-dockercfg-xvd7q\""
Nov 01 15:01:05.521451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.516705    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-8d6ea2\""
Nov 01 15:01:05.654434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.615928    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-kube-api-access-m9rqq\") pod \"payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f\" (UID: \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:05.654434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.616047    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f\" (UID: \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:05.654434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.616896    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f\" (UID: \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:05.654434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.626102    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"]
Nov 01 15:01:05.654434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.632034    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:05.659085 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.659055    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"]
Nov 01 15:01:05.675129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.675099    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-kube-api-access-m9rqq\") pod \"payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f\" (UID: \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:05.716936 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.716907    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e587e-b0d1-497a-8508-4ac760aebca1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6\" (UID: \"641e587e-b0d1-497a-8508-4ac760aebca1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:05.717155 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.717136    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhl5\" (UniqueName: \"kubernetes.io/projected/641e587e-b0d1-497a-8508-4ac760aebca1-kube-api-access-zzhl5\") pod \"payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6\" (UID: \"641e587e-b0d1-497a-8508-4ac760aebca1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:05.724737 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.724702    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:05.818692 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.818663    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhl5\" (UniqueName: \"kubernetes.io/projected/641e587e-b0d1-497a-8508-4ac760aebca1-kube-api-access-zzhl5\") pod \"payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6\" (UID: \"641e587e-b0d1-497a-8508-4ac760aebca1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:05.818857 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.818823    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e587e-b0d1-497a-8508-4ac760aebca1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6\" (UID: \"641e587e-b0d1-497a-8508-4ac760aebca1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:05.819633 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.819606    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e587e-b0d1-497a-8508-4ac760aebca1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6\" (UID: \"641e587e-b0d1-497a-8508-4ac760aebca1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:05.879929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.879900    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhl5\" (UniqueName: \"kubernetes.io/projected/641e587e-b0d1-497a-8508-4ac760aebca1-kube-api-access-zzhl5\") pod \"payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6\" (UID: \"641e587e-b0d1-497a-8508-4ac760aebca1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:05.976635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:05.976608    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:06.101423 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:01:06.101375    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f24afc_90c5_4b19_bea4_fd19a1e81cd1.slice/crio-9252de6cee85c6d6f7850cf0ae50245dca3cc046964b8b57ec49249922678424 WatchSource:0}: Error finding container 9252de6cee85c6d6f7850cf0ae50245dca3cc046964b8b57ec49249922678424: Status 404 returned error can't find the container with id 9252de6cee85c6d6f7850cf0ae50245dca3cc046964b8b57ec49249922678424
Nov 01 15:01:06.106444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:06.105238    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"]
Nov 01 15:01:06.347439 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:01:06.347405    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641e587e_b0d1_497a_8508_4ac760aebca1.slice/crio-08ac2f5c65ab7bbb191cd226f38fcf0b4db6b9b21284a4c3cd68a8432bd6f41d WatchSource:0}: Error finding container 08ac2f5c65ab7bbb191cd226f38fcf0b4db6b9b21284a4c3cd68a8432bd6f41d: Status 404 returned error can't find the container with id 08ac2f5c65ab7bbb191cd226f38fcf0b4db6b9b21284a4c3cd68a8432bd6f41d
Nov 01 15:01:06.350869 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:06.350847    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"]
Nov 01 15:01:07.064127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:07.064084    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6" event={"ID":"641e587e-b0d1-497a-8508-4ac760aebca1","Type":"ContainerStarted","Data":"391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8"}
Nov 01 15:01:07.064331 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:07.064142    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6" event={"ID":"641e587e-b0d1-497a-8508-4ac760aebca1","Type":"ContainerStarted","Data":"08ac2f5c65ab7bbb191cd226f38fcf0b4db6b9b21284a4c3cd68a8432bd6f41d"}
Nov 01 15:01:07.064331 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:07.064277    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:07.065669 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:07.065644    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f" event={"ID":"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1","Type":"ContainerStarted","Data":"504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191"}
Nov 01 15:01:07.065769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:07.065689    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f" event={"ID":"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1","Type":"ContainerStarted","Data":"9252de6cee85c6d6f7850cf0ae50245dca3cc046964b8b57ec49249922678424"}
Nov 01 15:01:07.065769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:07.065726    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:07.184518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:07.184464    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6" podStartSLOduration=2.184450384 podStartE2EDuration="2.184450384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:01:07.182024343 +0000 UTC m=+2607.347509644" watchObservedRunningTime="2026-11-01 15:01:07.184450384 +0000 UTC m=+2607.349935707"
Nov 01 15:01:07.314405 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:07.314275    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f" podStartSLOduration=2.314249839 podStartE2EDuration="2.314249839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:01:07.302601932 +0000 UTC m=+2607.468087230" watchObservedRunningTime="2026-11-01 15:01:07.314249839 +0000 UTC m=+2607.479735138"
Nov 01 15:01:11.022014 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:11.021984    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:01:18.076338 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:18.076307    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:01:18.076800 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:18.076380    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:01:26.050953 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:26.050866    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:01:34.249537 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:34.249496    5423 patch_prober.go:29] interesting pod/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:01:34.249950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:34.249553    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:01:35.150508 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:01:35.150465    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:01:37.281212 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:37.281179    5423 generic.go:356] "Generic (PLEG): container finished" podID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerID="e7a76aa55d023232c4068a3d9b883256ce26e23e3b7c26ca2ccc1348b6e0e882" exitCode=0
Nov 01 15:01:37.281686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:37.281242    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" event={"ID":"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154","Type":"ContainerDied","Data":"e7a76aa55d023232c4068a3d9b883256ce26e23e3b7c26ca2ccc1348b6e0e882"}
Nov 01 15:01:37.281686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:37.281675    5423 scope.go:122] "RemoveContainer" containerID="e7a76aa55d023232c4068a3d9b883256ce26e23e3b7c26ca2ccc1348b6e0e882"
Nov 01 15:01:37.283346 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:37.283309    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:01:37.283489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:37.283380    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:01:37.283489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:37.283428    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:01:38.327687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:38.327644    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" event={"ID":"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154","Type":"ContainerStarted","Data":"47dec0c674ed22714a6e580cd2e82a69a59b99b82f32bf93b6e46b99e2db10b7"}
Nov 01 15:01:38.519604 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:38.519551    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:01:38.520044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:38.520008    5423 patch_prober.go:29] interesting pod/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 15:01:38.520153 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:38.520076    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 15:01:39.520411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:39.519762    5423 patch_prober.go:29] interesting pod/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 15:01:39.520411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:39.519841    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 15:01:40.517764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:40.517731    5423 patch_prober.go:29] interesting pod/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 15:01:40.518059 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:40.518031    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 15:01:41.544141 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:41.544108    5423 patch_prober.go:29] interesting pod/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 15:01:41.544753 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:41.544721    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 15:01:42.515850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:42.515816    5423 patch_prober.go:29] interesting pod/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 15:01:42.516141 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:42.516110    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 15:01:43.520325 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:43.520294    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:01:43.520757 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:43.520617    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:01:43.521215 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:01:43.521197    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:02:05.148441 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:02:05.148377    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:02:09.468349 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.468315    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"]
Nov 01 15:02:09.471550 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.471530    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.512534 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.512496    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-home\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.512673 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.512585    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-tmp-dir\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.512752 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.512725    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f77ec289-210e-4b97-adca-ff89072202c9-tls-certs\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.512823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.512770    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-model-cache\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.512823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.512802    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-dshm\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.512934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.512866    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc922\" (UniqueName: \"kubernetes.io/projected/f77ec289-210e-4b97-adca-ff89072202c9-kube-api-access-mc922\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.576849 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.576821    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-6fcb7b\"/\"kube-root-ca.crt\""
Nov 01 15:02:09.577033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.576867    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-6fcb7b\"/\"autores-model-6fcb7b-kserve-self-signed-certs\""
Nov 01 15:02:09.577033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.576985    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-6fcb7b\"/\"default-dockercfg-cn2xs\""
Nov 01 15:02:09.577033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.577016    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-6fcb7b\"/\"openshift-service-ca.crt\""
Nov 01 15:02:09.579817 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.579793    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"]
Nov 01 15:02:09.637674 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.637635    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-home\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.637989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.637963    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-tmp-dir\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.638212 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.638194    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f77ec289-210e-4b97-adca-ff89072202c9-tls-certs\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.638342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.638327    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-model-cache\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.638492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.638474    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-dshm\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.638655 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.638637    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc922\" (UniqueName: \"kubernetes.io/projected/f77ec289-210e-4b97-adca-ff89072202c9-kube-api-access-mc922\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.639831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.639805    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-model-cache\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.646806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.646767    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-tmp-dir\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.654475 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.649408    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f77ec289-210e-4b97-adca-ff89072202c9-tls-certs\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.658494 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.657877    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-home\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.672910 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.672878    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-dshm\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.696927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.696893    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc922\" (UniqueName: \"kubernetes.io/projected/f77ec289-210e-4b97-adca-ff89072202c9-kube-api-access-mc922\") pod \"autores-model-6fcb7b-kserve-6f89d667f7-gl786\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") " pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:09.782754 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:09.782671    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:10.154937 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:02:10.154902    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf77ec289_210e_4b97_adca_ff89072202c9.slice/crio-e37efc061e895d81b53027e90efa071195dde5a0e7408132b31c0e9646db313a WatchSource:0}: Error finding container e37efc061e895d81b53027e90efa071195dde5a0e7408132b31c0e9646db313a: Status 404 returned error can't find the container with id e37efc061e895d81b53027e90efa071195dde5a0e7408132b31c0e9646db313a
Nov 01 15:02:10.158211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:10.158187    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"]
Nov 01 15:02:10.679651 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:10.679612    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786" event={"ID":"f77ec289-210e-4b97-adca-ff89072202c9","Type":"ContainerStarted","Data":"357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346"}
Nov 01 15:02:10.680044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:10.679663    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786" event={"ID":"f77ec289-210e-4b97-adca-ff89072202c9","Type":"ContainerStarted","Data":"e37efc061e895d81b53027e90efa071195dde5a0e7408132b31c0e9646db313a"}
Nov 01 15:02:10.783052 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:10.783019    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:10.796237 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:10.796187    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786" podStartSLOduration=1.7961743860000001 podStartE2EDuration="1.796174386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:02:10.795482436 +0000 UTC m=+2670.960967734" watchObservedRunningTime="2026-11-01 15:02:10.796174386 +0000 UTC m=+2670.961659708"
Nov 01 15:02:11.049662 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:11.049572    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:02:19.783252 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:19.783219    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:19.796519 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:19.796489    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:19.857202 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:19.857176    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:02:20.854121 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:20.854088    5423 generic.go:356] "Generic (PLEG): container finished" podID="0966ad2c-b364-4404-9fdc-479744097176" containerID="42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f" exitCode=0
Nov 01 15:02:20.854509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:20.854156    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" event={"ID":"0966ad2c-b364-4404-9fdc-479744097176","Type":"ContainerDied","Data":"42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f"}
Nov 01 15:02:20.854622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:20.854605    5423 scope.go:122] "RemoveContainer" containerID="42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f"
Nov 01 15:02:20.856266 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:20.856237    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:02:20.856340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:20.856324    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:02:20.856420 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:20.856374    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:02:21.864004 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:21.863970    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" event={"ID":"0966ad2c-b364-4404-9fdc-479744097176","Type":"ContainerStarted","Data":"368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1"}
Nov 01 15:02:22.567843 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:22.567810    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:02:22.569028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:22.568997    5423 patch_prober.go:29] interesting pod/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 15:02:22.569162 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:22.569063    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 15:02:22.699411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:22.695256    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:02:23.568475 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:23.568441    5423 patch_prober.go:29] interesting pod/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 15:02:23.568879 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:23.568493    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 15:02:24.567756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:24.567715    5423 patch_prober.go:29] interesting pod/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 15:02:24.567959 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:24.567781    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 15:02:25.567905 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:25.567864    5423 patch_prober.go:29] interesting pod/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 15:02:25.568305 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:25.567926    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 15:02:26.567854 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:26.567817    5423 patch_prober.go:29] interesting pod/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 15:02:26.568065 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:26.567879    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 15:02:27.572081 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:27.572051    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:02:27.573111 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:27.573084    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:02:35.150022 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:02:35.149986    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:02:42.548120 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:42.548091    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:02:42.581337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:42.581310    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:02:53.735881 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:02:53.735788    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:03:05.149795 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:03:05.149760    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:03:07.772508 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:07.772467    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"]
Nov 01 15:03:22.793326 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:22.793273    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786" podUID="f77ec289-210e-4b97-adca-ff89072202c9" containerName="main" containerID="cri-o://357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346" gracePeriod=45
Nov 01 15:03:22.845766 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:22.845725    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786" podUID="f77ec289-210e-4b97-adca-ff89072202c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": dial tcp 10.133.0.176:8000: connect: connection refused"
Nov 01 15:03:23.064213 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.064187    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:03:23.114106 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114073    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f77ec289-210e-4b97-adca-ff89072202c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f77ec289-210e-4b97-adca-ff89072202c9-tls-certs\") pod \"f77ec289-210e-4b97-adca-ff89072202c9\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") "
Nov 01 15:03:23.114217 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114148    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-tmp-dir\") pod \"f77ec289-210e-4b97-adca-ff89072202c9\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") "
Nov 01 15:03:23.114217 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114183    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-dshm\") pod \"f77ec289-210e-4b97-adca-ff89072202c9\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") "
Nov 01 15:03:23.114298 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114280    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f77ec289-210e-4b97-adca-ff89072202c9-kube-api-access-mc922\" (UniqueName: \"kubernetes.io/projected/f77ec289-210e-4b97-adca-ff89072202c9-kube-api-access-mc922\") pod \"f77ec289-210e-4b97-adca-ff89072202c9\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") "
Nov 01 15:03:23.114359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114319    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-model-cache\") pod \"f77ec289-210e-4b97-adca-ff89072202c9\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") "
Nov 01 15:03:23.114454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114424    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-tmp-dir" pod "f77ec289-210e-4b97-adca-ff89072202c9" (UID: "f77ec289-210e-4b97-adca-ff89072202c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:03:23.114522 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114440    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-home\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-home\") pod \"f77ec289-210e-4b97-adca-ff89072202c9\" (UID: \"f77ec289-210e-4b97-adca-ff89072202c9\") "
Nov 01 15:03:23.114684 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114663    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-home" pod "f77ec289-210e-4b97-adca-ff89072202c9" (UID: "f77ec289-210e-4b97-adca-ff89072202c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:03:23.114769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.114698    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-model-cache" pod "f77ec289-210e-4b97-adca-ff89072202c9" (UID: "f77ec289-210e-4b97-adca-ff89072202c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:03:23.115108 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.115084    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:23.115215 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.115114    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:23.115215 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.115130    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:23.116481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.116451    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-dshm" pod "f77ec289-210e-4b97-adca-ff89072202c9" (UID: "f77ec289-210e-4b97-adca-ff89072202c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:03:23.116579 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.116494    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77ec289-210e-4b97-adca-ff89072202c9-tls-certs" pod "f77ec289-210e-4b97-adca-ff89072202c9" (UID: "f77ec289-210e-4b97-adca-ff89072202c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:03:23.116579 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.116528    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77ec289-210e-4b97-adca-ff89072202c9-kube-api-access-mc922" pod "f77ec289-210e-4b97-adca-ff89072202c9" (UID: "f77ec289-210e-4b97-adca-ff89072202c9"). InnerVolumeSpecName "kube-api-access-mc922". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:03:23.216010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.215979    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f77ec289-210e-4b97-adca-ff89072202c9-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:23.216010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.216009    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f77ec289-210e-4b97-adca-ff89072202c9-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:23.216166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.216024    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc922\" (UniqueName: \"kubernetes.io/projected/f77ec289-210e-4b97-adca-ff89072202c9-kube-api-access-mc922\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:23.435635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.435597    5423 generic.go:356] "Generic (PLEG): container finished" podID="f77ec289-210e-4b97-adca-ff89072202c9" containerID="357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346" exitCode=0
Nov 01 15:03:23.435828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.435644    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"
Nov 01 15:03:23.435828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.435683    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786" event={"ID":"f77ec289-210e-4b97-adca-ff89072202c9","Type":"ContainerDied","Data":"357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346"}
Nov 01 15:03:23.435828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.435733    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786" event={"ID":"f77ec289-210e-4b97-adca-ff89072202c9","Type":"ContainerDied","Data":"e37efc061e895d81b53027e90efa071195dde5a0e7408132b31c0e9646db313a"}
Nov 01 15:03:23.435828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.435759    5423 scope.go:122] "RemoveContainer" containerID="357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346"
Nov 01 15:03:23.446650 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.446632    5423 scope.go:122] "RemoveContainer" containerID="357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346"
Nov 01 15:03:23.446882 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:03:23.446864    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346\": container with ID starting with 357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346 not found: ID does not exist" containerID="357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346"
Nov 01 15:03:23.446935 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.446892    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346"} err="failed to get container status \"357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346\": rpc error: code = NotFound desc = could not find container \"357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346\": container with ID starting with 357c09c6d1aa82cc10afea91ce24873e637f52868dc0890ca095b307a1da1346 not found: ID does not exist"
Nov 01 15:03:23.603112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.603083    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"]
Nov 01 15:03:23.606418 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.606393    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-6fcb7b/autores-model-6fcb7b-kserve-6f89d667f7-gl786"]
Nov 01 15:03:23.961886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:23.961849    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f77ec289-210e-4b97-adca-ff89072202c9" path="/var/lib/kubelet/pods/f77ec289-210e-4b97-adca-ff89072202c9/volumes"
Nov 01 15:03:25.828666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:25.828632    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"]
Nov 01 15:03:25.939397 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:25.939351    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"]
Nov 01 15:03:26.378032 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:26.377981    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:03:30.829808 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:30.829760    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk" podUID="86b4a585-907c-4d1f-a073-31087845f7e6" containerName="payload-processing" containerID="cri-o://764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe" gracePeriod=25
Nov 01 15:03:30.946412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:30.945463    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp" podUID="84836356-3ad8-4aca-bea9-ff472a135724" containerName="payload-pre-processing" containerID="cri-o://1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e" gracePeriod=25
Nov 01 15:03:31.154844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.154818    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 15:03:31.214561 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.214535    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 15:03:31.225630 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.225604    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b4a585-907c-4d1f-a073-31087845f7e6-kube-api-access-vpz2f\" (UniqueName: \"kubernetes.io/projected/86b4a585-907c-4d1f-a073-31087845f7e6-kube-api-access-vpz2f\") pod \"86b4a585-907c-4d1f-a073-31087845f7e6\" (UID: \"86b4a585-907c-4d1f-a073-31087845f7e6\") "
Nov 01 15:03:31.225772 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.225692    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b4a585-907c-4d1f-a073-31087845f7e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b4a585-907c-4d1f-a073-31087845f7e6-plugins-config-volume\") pod \"86b4a585-907c-4d1f-a073-31087845f7e6\" (UID: \"86b4a585-907c-4d1f-a073-31087845f7e6\") "
Nov 01 15:03:31.226095 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.226071    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b4a585-907c-4d1f-a073-31087845f7e6-plugins-config-volume" pod "86b4a585-907c-4d1f-a073-31087845f7e6" (UID: "86b4a585-907c-4d1f-a073-31087845f7e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:03:31.226328 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.226313    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b4a585-907c-4d1f-a073-31087845f7e6-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:31.227658 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.227637    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b4a585-907c-4d1f-a073-31087845f7e6-kube-api-access-vpz2f" pod "86b4a585-907c-4d1f-a073-31087845f7e6" (UID: "86b4a585-907c-4d1f-a073-31087845f7e6"). InnerVolumeSpecName "kube-api-access-vpz2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:03:31.327285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.327252    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84836356-3ad8-4aca-bea9-ff472a135724-kube-api-access-g7kgj\" (UniqueName: \"kubernetes.io/projected/84836356-3ad8-4aca-bea9-ff472a135724-kube-api-access-g7kgj\") pod \"84836356-3ad8-4aca-bea9-ff472a135724\" (UID: \"84836356-3ad8-4aca-bea9-ff472a135724\") "
Nov 01 15:03:31.327470 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.327422    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84836356-3ad8-4aca-bea9-ff472a135724-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84836356-3ad8-4aca-bea9-ff472a135724-plugins-config-volume\") pod \"84836356-3ad8-4aca-bea9-ff472a135724\" (UID: \"84836356-3ad8-4aca-bea9-ff472a135724\") "
Nov 01 15:03:31.327801 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.327776    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpz2f\" (UniqueName: \"kubernetes.io/projected/86b4a585-907c-4d1f-a073-31087845f7e6-kube-api-access-vpz2f\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:31.327801 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.327778    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84836356-3ad8-4aca-bea9-ff472a135724-plugins-config-volume" pod "84836356-3ad8-4aca-bea9-ff472a135724" (UID: "84836356-3ad8-4aca-bea9-ff472a135724"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:03:31.329342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.329322    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84836356-3ad8-4aca-bea9-ff472a135724-kube-api-access-g7kgj" pod "84836356-3ad8-4aca-bea9-ff472a135724" (UID: "84836356-3ad8-4aca-bea9-ff472a135724"). InnerVolumeSpecName "kube-api-access-g7kgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:03:31.429201 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.429114    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7kgj\" (UniqueName: \"kubernetes.io/projected/84836356-3ad8-4aca-bea9-ff472a135724-kube-api-access-g7kgj\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:31.429201 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.429148    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84836356-3ad8-4aca-bea9-ff472a135724-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:31.498423 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.498371    5423 generic.go:356] "Generic (PLEG): container finished" podID="86b4a585-907c-4d1f-a073-31087845f7e6" containerID="764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe" exitCode=0
Nov 01 15:03:31.498607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.498440    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"
Nov 01 15:03:31.498607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.498455    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk" event={"ID":"86b4a585-907c-4d1f-a073-31087845f7e6","Type":"ContainerDied","Data":"764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe"}
Nov 01 15:03:31.498607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.498529    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk" event={"ID":"86b4a585-907c-4d1f-a073-31087845f7e6","Type":"ContainerDied","Data":"89a554df7f31faf7d74c831d13c8644a6c0b4a110b90e60399afd7b833c4077c"}
Nov 01 15:03:31.498607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.498559    5423 scope.go:122] "RemoveContainer" containerID="764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe"
Nov 01 15:03:31.499958 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.499935    5423 generic.go:356] "Generic (PLEG): container finished" podID="84836356-3ad8-4aca-bea9-ff472a135724" containerID="1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e" exitCode=0
Nov 01 15:03:31.500076 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.499997    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"
Nov 01 15:03:31.500076 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.500023    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp" event={"ID":"84836356-3ad8-4aca-bea9-ff472a135724","Type":"ContainerDied","Data":"1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e"}
Nov 01 15:03:31.500169 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.500074    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp" event={"ID":"84836356-3ad8-4aca-bea9-ff472a135724","Type":"ContainerDied","Data":"87cc1fb13e45da9ae9efb62f71d940c8704fc806e7844ad9ca3cbd598f4d636a"}
Nov 01 15:03:31.510639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.510621    5423 scope.go:122] "RemoveContainer" containerID="764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe"
Nov 01 15:03:31.510909 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:03:31.510890    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe\": container with ID starting with 764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe not found: ID does not exist" containerID="764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe"
Nov 01 15:03:31.510980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.510917    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe"} err="failed to get container status \"764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe\": rpc error: code = NotFound desc = could not find container \"764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe\": container with ID starting with 764ced8816118a4225e971507caa0ab135d608a3b022193258448e66d8c5aafe not found: ID does not exist"
Nov 01 15:03:31.510980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.510943    5423 scope.go:122] "RemoveContainer" containerID="1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e"
Nov 01 15:03:31.520617 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.520599    5423 scope.go:122] "RemoveContainer" containerID="1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e"
Nov 01 15:03:31.520824 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:03:31.520805    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e\": container with ID starting with 1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e not found: ID does not exist" containerID="1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e"
Nov 01 15:03:31.520882 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.520832    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e"} err="failed to get container status \"1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e\": rpc error: code = NotFound desc = could not find container \"1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e\": container with ID starting with 1500254b2d2d92ee3348b9bed9fe1970b73ee79e73148330ac78d799ef98aa0e not found: ID does not exist"
Nov 01 15:03:31.669967 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.669933    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"]
Nov 01 15:03:31.675624 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.675598    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c9d0a-66cb68d84f-4mqlk"]
Nov 01 15:03:31.787795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.787722    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"]
Nov 01 15:03:31.791648 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.791624    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c9d0a-6d4d556d75-nvpcp"]
Nov 01 15:03:31.961306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.961274    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84836356-3ad8-4aca-bea9-ff472a135724" path="/var/lib/kubelet/pods/84836356-3ad8-4aca-bea9-ff472a135724/volumes"
Nov 01 15:03:31.961736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:31.961720    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b4a585-907c-4d1f-a073-31087845f7e6" path="/var/lib/kubelet/pods/86b4a585-907c-4d1f-a073-31087845f7e6/volumes"
Nov 01 15:03:35.149673 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:03:35.149640    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:03:46.884502 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.884464    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"]
Nov 01 15:03:46.885094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885080    5423 cpu_manager.go:414] "containerMap: removing container" podUID="86b4a585-907c-4d1f-a073-31087845f7e6" containerName="payload-processing"
Nov 01 15:03:46.885143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885096    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b4a585-907c-4d1f-a073-31087845f7e6" containerName="payload-processing"
Nov 01 15:03:46.885143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885115    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f77ec289-210e-4b97-adca-ff89072202c9" containerName="main"
Nov 01 15:03:46.885143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885121    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f77ec289-210e-4b97-adca-ff89072202c9" containerName="main"
Nov 01 15:03:46.885143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885131    5423 cpu_manager.go:414] "containerMap: removing container" podUID="84836356-3ad8-4aca-bea9-ff472a135724" containerName="payload-pre-processing"
Nov 01 15:03:46.885143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885137    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84836356-3ad8-4aca-bea9-ff472a135724" containerName="payload-pre-processing"
Nov 01 15:03:46.885298 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885255    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b4a585-907c-4d1f-a073-31087845f7e6" containerName="payload-processing"
Nov 01 15:03:46.885298 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885265    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f77ec289-210e-4b97-adca-ff89072202c9" containerName="main"
Nov 01 15:03:46.885298 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.885273    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="84836356-3ad8-4aca-bea9-ff472a135724" containerName="payload-pre-processing"
Nov 01 15:03:46.888211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.888196    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:46.976006 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.975974    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"]
Nov 01 15:03:46.992876 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.992846    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-97c87c\"/\"openshift-service-ca.crt\""
Nov 01 15:03:46.993022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.992895    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-97c87c\"/\"default-dockercfg-pbwqb\""
Nov 01 15:03:46.993022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.992952    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-97c87c\"/\"auth-test-model-97c87c-kserve-self-signed-certs\""
Nov 01 15:03:46.993022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:46.993002    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-97c87c\"/\"kube-root-ca.crt\""
Nov 01 15:03:47.022067 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.022038    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-dshm\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.022179 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.022145    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81550cdb-ffc0-4721-8d0f-fa875acb002d-tls-certs\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.022243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.022182    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggt2l\" (UniqueName: \"kubernetes.io/projected/81550cdb-ffc0-4721-8d0f-fa875acb002d-kube-api-access-ggt2l\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.022297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.022283    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-model-cache\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.022336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.022327    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-home\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.022474 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.022455    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-tmp-dir\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.124223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.124189    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-home\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.124435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.124250    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-tmp-dir\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.124531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.124428    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-dshm\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.124723 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.124697    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-home\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.124847 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.124740    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-tmp-dir\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.124901 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.124848    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81550cdb-ffc0-4721-8d0f-fa875acb002d-tls-certs\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.124901 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.124889    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggt2l\" (UniqueName: \"kubernetes.io/projected/81550cdb-ffc0-4721-8d0f-fa875acb002d-kube-api-access-ggt2l\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.125007 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.124988    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-model-cache\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.125446 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.125421    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-model-cache\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.126935 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.126911    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-dshm\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.127363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.127338    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81550cdb-ffc0-4721-8d0f-fa875acb002d-tls-certs\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.183974 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.183912    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggt2l\" (UniqueName: \"kubernetes.io/projected/81550cdb-ffc0-4721-8d0f-fa875acb002d-kube-api-access-ggt2l\") pod \"auth-test-model-97c87c-kserve-54775d54cc-84k7c\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") " pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.199577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.199559    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:47.571263 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:03:47.571233    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81550cdb_ffc0_4721_8d0f_fa875acb002d.slice/crio-d49cdec5740c96f12c02e487dd2be1730a644ca3206853f4f9838ea0c0185aaa WatchSource:0}: Error finding container d49cdec5740c96f12c02e487dd2be1730a644ca3206853f4f9838ea0c0185aaa: Status 404 returned error can't find the container with id d49cdec5740c96f12c02e487dd2be1730a644ca3206853f4f9838ea0c0185aaa
Nov 01 15:03:47.573991 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.573967    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"]
Nov 01 15:03:47.605315 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:47.605289    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c" event={"ID":"81550cdb-ffc0-4721-8d0f-fa875acb002d","Type":"ContainerStarted","Data":"d49cdec5740c96f12c02e487dd2be1730a644ca3206853f4f9838ea0c0185aaa"}
Nov 01 15:03:48.613071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:48.613040    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c" event={"ID":"81550cdb-ffc0-4721-8d0f-fa875acb002d","Type":"ContainerStarted","Data":"e94ed62b18d8d37c666c638580ea89b0e2974d935566896a097feb1594a41878"}
Nov 01 15:03:48.730536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:48.730488    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c" podStartSLOduration=2.730474314 podStartE2EDuration="2.730474314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:03:48.729686884 +0000 UTC m=+2768.895172182" watchObservedRunningTime="2026-11-01 15:03:48.730474314 +0000 UTC m=+2768.895959612"
Nov 01 15:03:49.098773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:49.098734    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"]
Nov 01 15:03:49.199540 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:49.199505    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"]
Nov 01 15:03:49.199734 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:49.199698    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:54.099424 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.099359    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2" podUID="3ca03e2f-f398-4a2c-b1ce-f86620269544" containerName="payload-processing" containerID="cri-o://1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa" gracePeriod=25
Nov 01 15:03:54.199780 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.199737    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt" podUID="496ebcf6-2c66-409a-8d43-561bf1e1118d" containerName="payload-pre-processing" containerID="cri-o://c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50" gracePeriod=25
Nov 01 15:03:54.368630 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.368571    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:03:54.492767 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.492743    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:03:54.522230 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.522192    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca03e2f-f398-4a2c-b1ce-f86620269544-kube-api-access-59pcs\" (UniqueName: \"kubernetes.io/projected/3ca03e2f-f398-4a2c-b1ce-f86620269544-kube-api-access-59pcs\") pod \"3ca03e2f-f398-4a2c-b1ce-f86620269544\" (UID: \"3ca03e2f-f398-4a2c-b1ce-f86620269544\") "
Nov 01 15:03:54.522373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.522352    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca03e2f-f398-4a2c-b1ce-f86620269544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca03e2f-f398-4a2c-b1ce-f86620269544-plugins-config-volume\") pod \"3ca03e2f-f398-4a2c-b1ce-f86620269544\" (UID: \"3ca03e2f-f398-4a2c-b1ce-f86620269544\") "
Nov 01 15:03:54.522803 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.522772    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca03e2f-f398-4a2c-b1ce-f86620269544-plugins-config-volume" pod "3ca03e2f-f398-4a2c-b1ce-f86620269544" (UID: "3ca03e2f-f398-4a2c-b1ce-f86620269544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:03:54.522973 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.522952    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca03e2f-f398-4a2c-b1ce-f86620269544-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:54.524489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.524461    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca03e2f-f398-4a2c-b1ce-f86620269544-kube-api-access-59pcs" pod "3ca03e2f-f398-4a2c-b1ce-f86620269544" (UID: "3ca03e2f-f398-4a2c-b1ce-f86620269544"). InnerVolumeSpecName "kube-api-access-59pcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:03:54.624277 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.624201    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496ebcf6-2c66-409a-8d43-561bf1e1118d-kube-api-access-9k4pn\" (UniqueName: \"kubernetes.io/projected/496ebcf6-2c66-409a-8d43-561bf1e1118d-kube-api-access-9k4pn\") pod \"496ebcf6-2c66-409a-8d43-561bf1e1118d\" (UID: \"496ebcf6-2c66-409a-8d43-561bf1e1118d\") "
Nov 01 15:03:54.624472 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.624296    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496ebcf6-2c66-409a-8d43-561bf1e1118d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496ebcf6-2c66-409a-8d43-561bf1e1118d-plugins-config-volume\") pod \"496ebcf6-2c66-409a-8d43-561bf1e1118d\" (UID: \"496ebcf6-2c66-409a-8d43-561bf1e1118d\") "
Nov 01 15:03:54.624775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.624738    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496ebcf6-2c66-409a-8d43-561bf1e1118d-plugins-config-volume" pod "496ebcf6-2c66-409a-8d43-561bf1e1118d" (UID: "496ebcf6-2c66-409a-8d43-561bf1e1118d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:03:54.625118 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.625095    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496ebcf6-2c66-409a-8d43-561bf1e1118d-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:54.625209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.625122    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59pcs\" (UniqueName: \"kubernetes.io/projected/3ca03e2f-f398-4a2c-b1ce-f86620269544-kube-api-access-59pcs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:54.626360 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.626336    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496ebcf6-2c66-409a-8d43-561bf1e1118d-kube-api-access-9k4pn" pod "496ebcf6-2c66-409a-8d43-561bf1e1118d" (UID: "496ebcf6-2c66-409a-8d43-561bf1e1118d"). InnerVolumeSpecName "kube-api-access-9k4pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:03:54.684787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.684752    5423 generic.go:356] "Generic (PLEG): container finished" podID="496ebcf6-2c66-409a-8d43-561bf1e1118d" containerID="c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50" exitCode=0
Nov 01 15:03:54.684945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.684797    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"
Nov 01 15:03:54.684945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.684830    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt" event={"ID":"496ebcf6-2c66-409a-8d43-561bf1e1118d","Type":"ContainerDied","Data":"c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50"}
Nov 01 15:03:54.684945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.684882    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt" event={"ID":"496ebcf6-2c66-409a-8d43-561bf1e1118d","Type":"ContainerDied","Data":"0400c9c1321293f6830730a12cefa04408a300d5c0252b29e74799ee44f1a6f9"}
Nov 01 15:03:54.684945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.684903    5423 scope.go:122] "RemoveContainer" containerID="c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50"
Nov 01 15:03:54.686358 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.686334    5423 generic.go:356] "Generic (PLEG): container finished" podID="3ca03e2f-f398-4a2c-b1ce-f86620269544" containerID="1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa" exitCode=0
Nov 01 15:03:54.686494 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.686378    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"
Nov 01 15:03:54.686494 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.686434    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2" event={"ID":"3ca03e2f-f398-4a2c-b1ce-f86620269544","Type":"ContainerDied","Data":"1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa"}
Nov 01 15:03:54.686494 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.686482    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2" event={"ID":"3ca03e2f-f398-4a2c-b1ce-f86620269544","Type":"ContainerDied","Data":"721e79688a0885058d645ee621d5219138faaf12b527d86cfdb1e908580e55c9"}
Nov 01 15:03:54.697341 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.697325    5423 scope.go:122] "RemoveContainer" containerID="c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50"
Nov 01 15:03:54.697634 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:03:54.697612    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50\": container with ID starting with c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50 not found: ID does not exist" containerID="c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50"
Nov 01 15:03:54.697700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.697642    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50"} err="failed to get container status \"c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50\": rpc error: code = NotFound desc = could not find container \"c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50\": container with ID starting with c57a92f7b847fa47ce50093c8824216a72373bb22ab9d7ca494204c2dd469d50 not found: ID does not exist"
Nov 01 15:03:54.697700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.697658    5423 scope.go:122] "RemoveContainer" containerID="1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa"
Nov 01 15:03:54.709418 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.709406    5423 scope.go:122] "RemoveContainer" containerID="1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa"
Nov 01 15:03:54.710110 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:03:54.710086    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa\": container with ID starting with 1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa not found: ID does not exist" containerID="1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa"
Nov 01 15:03:54.710191 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.710118    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa"} err="failed to get container status \"1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa\": rpc error: code = NotFound desc = could not find container \"1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa\": container with ID starting with 1b0a2850d77eec24d0d7daafa73dc4dfb65c61bacc9b75ad1853f5fc0b79abaa not found: ID does not exist"
Nov 01 15:03:54.726635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.726608    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k4pn\" (UniqueName: \"kubernetes.io/projected/496ebcf6-2c66-409a-8d43-561bf1e1118d-kube-api-access-9k4pn\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:03:54.854654 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.854614    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"]
Nov 01 15:03:54.857063 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.857043    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6fcb7b-67b8c689b5-qrxz2"]
Nov 01 15:03:54.978261 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.978174    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"]
Nov 01 15:03:54.994314 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:54.994272    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6fcb7b-7db86d97bc-4fpqt"]
Nov 01 15:03:55.961307 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:55.961277    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca03e2f-f398-4a2c-b1ce-f86620269544" path="/var/lib/kubelet/pods/3ca03e2f-f398-4a2c-b1ce-f86620269544/volumes"
Nov 01 15:03:55.961695 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:55.961684    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496ebcf6-2c66-409a-8d43-561bf1e1118d" path="/var/lib/kubelet/pods/496ebcf6-2c66-409a-8d43-561bf1e1118d/volumes"
Nov 01 15:03:57.200425 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:57.200369    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:57.212620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:57.212591    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:57.719731 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:57.719704    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:03:59.383501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:03:59.383466    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:04:05.153900 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:04:05.153855    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:04:13.155361 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:13.155325    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"]
Nov 01 15:04:13.263180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:13.263149    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"]
Nov 01 15:04:18.156806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.156732    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6" podUID="641e587e-b0d1-497a-8508-4ac760aebca1" containerName="payload-processing" containerID="cri-o://391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8" gracePeriod=25
Nov 01 15:04:18.264019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.263951    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f" podUID="a3f24afc-90c5-4b19-bea4-fd19a1e81cd1" containerName="payload-pre-processing" containerID="cri-o://504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191" gracePeriod=25
Nov 01 15:04:18.462288 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.462219    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:04:18.566408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.566361    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:04:18.579298 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.579273    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641e587e-b0d1-497a-8508-4ac760aebca1-kube-api-access-zzhl5\" (UniqueName: \"kubernetes.io/projected/641e587e-b0d1-497a-8508-4ac760aebca1-kube-api-access-zzhl5\") pod \"641e587e-b0d1-497a-8508-4ac760aebca1\" (UID: \"641e587e-b0d1-497a-8508-4ac760aebca1\") "
Nov 01 15:04:18.579672 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.579645    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641e587e-b0d1-497a-8508-4ac760aebca1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e587e-b0d1-497a-8508-4ac760aebca1-plugins-config-volume\") pod \"641e587e-b0d1-497a-8508-4ac760aebca1\" (UID: \"641e587e-b0d1-497a-8508-4ac760aebca1\") "
Nov 01 15:04:18.580407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.580048    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641e587e-b0d1-497a-8508-4ac760aebca1-plugins-config-volume" pod "641e587e-b0d1-497a-8508-4ac760aebca1" (UID: "641e587e-b0d1-497a-8508-4ac760aebca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:18.580407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.580294    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e587e-b0d1-497a-8508-4ac760aebca1-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:18.605426 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.604558    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641e587e-b0d1-497a-8508-4ac760aebca1-kube-api-access-zzhl5" pod "641e587e-b0d1-497a-8508-4ac760aebca1" (UID: "641e587e-b0d1-497a-8508-4ac760aebca1"). InnerVolumeSpecName "kube-api-access-zzhl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:18.681623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.681596    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-kube-api-access-m9rqq\") pod \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\" (UID: \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\") "
Nov 01 15:04:18.681787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.681672    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-plugins-config-volume\") pod \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\" (UID: \"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1\") "
Nov 01 15:04:18.681980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.681962    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhl5\" (UniqueName: \"kubernetes.io/projected/641e587e-b0d1-497a-8508-4ac760aebca1-kube-api-access-zzhl5\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:18.682053 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.681995    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-plugins-config-volume" pod "a3f24afc-90c5-4b19-bea4-fd19a1e81cd1" (UID: "a3f24afc-90c5-4b19-bea4-fd19a1e81cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:18.683700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.683677    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-kube-api-access-m9rqq" pod "a3f24afc-90c5-4b19-bea4-fd19a1e81cd1" (UID: "a3f24afc-90c5-4b19-bea4-fd19a1e81cd1"). InnerVolumeSpecName "kube-api-access-m9rqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:18.783975 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.783898    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9rqq\" (UniqueName: \"kubernetes.io/projected/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-kube-api-access-m9rqq\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:18.783975 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.783933    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:18.801913 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.801874    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"]
Nov 01 15:04:18.802209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.802178    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy" containerID="cri-o://47dec0c674ed22714a6e580cd2e82a69a59b99b82f32bf93b6e46b99e2db10b7" gracePeriod=30
Nov 01 15:04:18.887659 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.887622    5423 generic.go:356] "Generic (PLEG): container finished" podID="641e587e-b0d1-497a-8508-4ac760aebca1" containerID="391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8" exitCode=0
Nov 01 15:04:18.887822 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.887678    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"
Nov 01 15:04:18.887822 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.887716    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6" event={"ID":"641e587e-b0d1-497a-8508-4ac760aebca1","Type":"ContainerDied","Data":"391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8"}
Nov 01 15:04:18.887822 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.887770    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6" event={"ID":"641e587e-b0d1-497a-8508-4ac760aebca1","Type":"ContainerDied","Data":"08ac2f5c65ab7bbb191cd226f38fcf0b4db6b9b21284a4c3cd68a8432bd6f41d"}
Nov 01 15:04:18.887822 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.887796    5423 scope.go:122] "RemoveContainer" containerID="391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8"
Nov 01 15:04:18.889546 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.889521    5423 generic.go:356] "Generic (PLEG): container finished" podID="a3f24afc-90c5-4b19-bea4-fd19a1e81cd1" containerID="504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191" exitCode=0
Nov 01 15:04:18.889694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.889567    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"
Nov 01 15:04:18.889694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.889588    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f" event={"ID":"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1","Type":"ContainerDied","Data":"504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191"}
Nov 01 15:04:18.889694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.889632    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f" event={"ID":"a3f24afc-90c5-4b19-bea4-fd19a1e81cd1","Type":"ContainerDied","Data":"9252de6cee85c6d6f7850cf0ae50245dca3cc046964b8b57ec49249922678424"}
Nov 01 15:04:18.897990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.897967    5423 scope.go:122] "RemoveContainer" containerID="391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8"
Nov 01 15:04:18.898292 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:04:18.898265    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8\": container with ID starting with 391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8 not found: ID does not exist" containerID="391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8"
Nov 01 15:04:18.898374 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.898303    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8"} err="failed to get container status \"391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8\": rpc error: code = NotFound desc = could not find container \"391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8\": container with ID starting with 391c6828d35abea10611799737a4ceb9a90cbbf121ec749f1a8cf610a1ab41a8 not found: ID does not exist"
Nov 01 15:04:18.898374 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.898324    5423 scope.go:122] "RemoveContainer" containerID="504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191"
Nov 01 15:04:18.907156 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.907140    5423 scope.go:122] "RemoveContainer" containerID="504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191"
Nov 01 15:04:18.907434 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:04:18.907415    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191\": container with ID starting with 504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191 not found: ID does not exist" containerID="504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191"
Nov 01 15:04:18.907496 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:18.907440    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191"} err="failed to get container status \"504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191\": rpc error: code = NotFound desc = could not find container \"504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191\": container with ID starting with 504cff277e6f0eb59c63e424c168a4e4ebc3a0f6d50110603e5858d399571191 not found: ID does not exist"
Nov 01 15:04:19.045703 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:19.045608    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"]
Nov 01 15:04:19.056270 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:19.048557    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8d6ea2-5bf9bcbdbc-5pq4f"]
Nov 01 15:04:19.154880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:19.154846    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"]
Nov 01 15:04:19.158295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:19.158269    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8d6ea2-649df5d9b5-djnw6"]
Nov 01 15:04:19.961944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:19.961850    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641e587e-b0d1-497a-8508-4ac760aebca1" path="/var/lib/kubelet/pods/641e587e-b0d1-497a-8508-4ac760aebca1/volumes"
Nov 01 15:04:19.962505 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:19.962479    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f24afc-90c5-4b19-bea4-fd19a1e81cd1" path="/var/lib/kubelet/pods/a3f24afc-90c5-4b19-bea4-fd19a1e81cd1/volumes"
Nov 01 15:04:23.963757 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:23.963720    5423 generic.go:356] "Generic (PLEG): container finished" podID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerID="47dec0c674ed22714a6e580cd2e82a69a59b99b82f32bf93b6e46b99e2db10b7" exitCode=0
Nov 01 15:04:23.964265 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:23.963924    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" event={"ID":"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154","Type":"ContainerDied","Data":"47dec0c674ed22714a6e580cd2e82a69a59b99b82f32bf93b6e46b99e2db10b7"}
Nov 01 15:04:23.964265 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:23.963985    5423 scope.go:122] "RemoveContainer" containerID="e7a76aa55d023232c4068a3d9b883256ce26e23e3b7c26ca2ccc1348b6e0e882"
Nov 01 15:04:24.079669 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.079607    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:04:24.180449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180419    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-certs\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.180449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180461    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-kube-api-access-rzqmb\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-kube-api-access-rzqmb\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.180794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180486    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-data\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.180794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180617    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-credential-socket\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.180794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180649    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-podinfo\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.180794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180729    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istiod-ca-cert\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.180794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180743    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-certs" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:24.180794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180757    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-socket\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.181129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180848    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-envoy\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.181129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.180984    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-socket" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:24.181129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181014    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-data" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:24.181300 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181153    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-credential-socket" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:24.181300 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181240    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-token\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-token\") pod \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\" (UID: \"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154\") "
Nov 01 15:04:24.181428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181304    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istiod-ca-cert" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:24.181878 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181851    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.181988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181883    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.181988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181898    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.181988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181910    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.181988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.181925    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.182934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.182911    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-kube-api-access-rzqmb" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "kube-api-access-rzqmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:24.183049 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.182951    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-podinfo" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:04:24.183313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.183296    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-envoy" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:24.183652 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.183638    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-token" pod "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" (UID: "e78e9ac0-aaa0-4a41-8122-f7dc9bb49154"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:24.282861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.282827    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzqmb\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-kube-api-access-rzqmb\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.282861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.282864    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.283068 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.282881    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.283068 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.282897    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:24.972329 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.972301    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"
Nov 01 15:04:24.972746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.972328    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q" event={"ID":"e78e9ac0-aaa0-4a41-8122-f7dc9bb49154","Type":"ContainerDied","Data":"39bb4c22ae133488f7d30b1aaebe571d2abbdd4a4cab39c924764aedad45e9fa"}
Nov 01 15:04:24.972746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:24.972395    5423 scope.go:122] "RemoveContainer" containerID="47dec0c674ed22714a6e580cd2e82a69a59b99b82f32bf93b6e46b99e2db10b7"
Nov 01 15:04:25.141885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.141855    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"]
Nov 01 15:04:25.193047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.193016    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-6fcb7b-openshift-default-5cd98976c9-sw72q"]
Nov 01 15:04:25.602929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.602900    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"]
Nov 01 15:04:25.603467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603450    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy"
Nov 01 15:04:25.603524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603468    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy"
Nov 01 15:04:25.603524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603481    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy"
Nov 01 15:04:25.603524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603487    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy"
Nov 01 15:04:25.603524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603497    5423 cpu_manager.go:414] "containerMap: removing container" podUID="496ebcf6-2c66-409a-8d43-561bf1e1118d" containerName="payload-pre-processing"
Nov 01 15:04:25.603524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603503    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496ebcf6-2c66-409a-8d43-561bf1e1118d" containerName="payload-pre-processing"
Nov 01 15:04:25.603524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603517    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3ca03e2f-f398-4a2c-b1ce-f86620269544" containerName="payload-processing"
Nov 01 15:04:25.603524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603522    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca03e2f-f398-4a2c-b1ce-f86620269544" containerName="payload-processing"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603537    5423 cpu_manager.go:414] "containerMap: removing container" podUID="a3f24afc-90c5-4b19-bea4-fd19a1e81cd1" containerName="payload-pre-processing"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603542    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f24afc-90c5-4b19-bea4-fd19a1e81cd1" containerName="payload-pre-processing"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603555    5423 cpu_manager.go:414] "containerMap: removing container" podUID="641e587e-b0d1-497a-8508-4ac760aebca1" containerName="payload-processing"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603559    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641e587e-b0d1-497a-8508-4ac760aebca1" containerName="payload-processing"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603659    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603666    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" containerName="istio-proxy"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603672    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="496ebcf6-2c66-409a-8d43-561bf1e1118d" containerName="payload-pre-processing"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603680    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca03e2f-f398-4a2c-b1ce-f86620269544" containerName="payload-processing"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603689    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f24afc-90c5-4b19-bea4-fd19a1e81cd1" containerName="payload-pre-processing"
Nov 01 15:04:25.603763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.603696    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="641e587e-b0d1-497a-8508-4ac760aebca1" containerName="payload-processing"
Nov 01 15:04:25.608490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.608470    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.662555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.662251    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ca0f9e\"/\"openshift-service-ca.crt\""
Nov 01 15:04:25.662555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.662371    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ca0f9e\"/\"kube-root-ca.crt\""
Nov 01 15:04:25.666691 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.666660    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"]
Nov 01 15:04:25.704478 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.704449    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-model-cache\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.704650 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.704628    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae153428-61eb-4192-842e-74cb00137fa6-tls-certs\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.704712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.704667    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-home\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.704712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.704696    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqlq\" (UniqueName: \"kubernetes.io/projected/ae153428-61eb-4192-842e-74cb00137fa6-kube-api-access-vmqlq\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.704807 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.704753    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-dshm\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.704858 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.704829    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-tmp-dir\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.712650 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.712627    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ca0f9e\"/\"default-dockercfg-p4lz5\""
Nov 01 15:04:25.712650 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.712635    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ca0f9e\"/\"auth-test-model-ca0f9e-kserve-self-signed-certs\""
Nov 01 15:04:25.806827 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.806785    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae153428-61eb-4192-842e-74cb00137fa6-tls-certs\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.807020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.806846    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-home\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.807020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.806895    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqlq\" (UniqueName: \"kubernetes.io/projected/ae153428-61eb-4192-842e-74cb00137fa6-kube-api-access-vmqlq\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.807020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.806924    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-dshm\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.807020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.806971    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-tmp-dir\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.807165 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.807061    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-model-cache\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.807522 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.807499    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-model-cache\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.807801 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.807779    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-home\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.808053 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.808031    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-tmp-dir\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.809895 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.809868    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-dshm\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.810060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.810039    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae153428-61eb-4192-842e-74cb00137fa6-tls-certs\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.865830 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.865762    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqlq\" (UniqueName: \"kubernetes.io/projected/ae153428-61eb-4192-842e-74cb00137fa6-kube-api-access-vmqlq\") pod \"auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.921221 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.921188    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:25.962063 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:25.962031    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78e9ac0-aaa0-4a41-8122-f7dc9bb49154" path="/var/lib/kubelet/pods/e78e9ac0-aaa0-4a41-8122-f7dc9bb49154/volumes"
Nov 01 15:04:26.300536 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:04:26.300503    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae153428_61eb_4192_842e_74cb00137fa6.slice/crio-62478eb0fdb4a1fe9121179cb6c33a45befb91c8d59bedf8ede21f691ad92aeb WatchSource:0}: Error finding container 62478eb0fdb4a1fe9121179cb6c33a45befb91c8d59bedf8ede21f691ad92aeb: Status 404 returned error can't find the container with id 62478eb0fdb4a1fe9121179cb6c33a45befb91c8d59bedf8ede21f691ad92aeb
Nov 01 15:04:26.303313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:26.303200    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"]
Nov 01 15:04:26.994033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:26.993998    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq" event={"ID":"ae153428-61eb-4192-842e-74cb00137fa6","Type":"ContainerStarted","Data":"3cd9b7e4df4d1d7a5d1504eb10dfc2d485556db32f10f326ad2058a89e798e40"}
Nov 01 15:04:26.994242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:26.994042    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq" event={"ID":"ae153428-61eb-4192-842e-74cb00137fa6","Type":"ContainerStarted","Data":"62478eb0fdb4a1fe9121179cb6c33a45befb91c8d59bedf8ede21f691ad92aeb"}
Nov 01 15:04:27.111187 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:27.111130    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq" podStartSLOduration=2.111116305 podStartE2EDuration="2.111116305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:04:27.109188029 +0000 UTC m=+2807.274673328" watchObservedRunningTime="2026-11-01 15:04:27.111116305 +0000 UTC m=+2807.276601603"
Nov 01 15:04:27.921848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:27.921820    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:28.331432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:28.331398    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"]
Nov 01 15:04:28.331668 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:28.331628    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy" containerID="cri-o://544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1" gracePeriod=30
Nov 01 15:04:29.451634 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.451601    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"]
Nov 01 15:04:29.455982 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.455961    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:29.515785 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.515747    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"]
Nov 01 15:04:29.580419 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.560293    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7794bf\""
Nov 01 15:04:29.580419 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.569598    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7794bf-dockercfg-nfbgq\""
Nov 01 15:04:29.604417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.582259    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921fae53-caf9-40e6-88f4-6ba8d6996f3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv\" (UID: \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:29.626723 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.626675    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd728\" (UniqueName: \"kubernetes.io/projected/921fae53-caf9-40e6-88f4-6ba8d6996f3c-kube-api-access-dd728\") pod \"payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv\" (UID: \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:29.734476 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.731799    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921fae53-caf9-40e6-88f4-6ba8d6996f3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv\" (UID: \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:29.734476 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.731911    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd728\" (UniqueName: \"kubernetes.io/projected/921fae53-caf9-40e6-88f4-6ba8d6996f3c-kube-api-access-dd728\") pod \"payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv\" (UID: \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:29.747654 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.747614    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921fae53-caf9-40e6-88f4-6ba8d6996f3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv\" (UID: \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:29.810227 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.810163    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd728\" (UniqueName: \"kubernetes.io/projected/921fae53-caf9-40e6-88f4-6ba8d6996f3c-kube-api-access-dd728\") pod \"payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv\" (UID: \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:29.978438 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.978406    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"]
Nov 01 15:04:29.978635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:29.978610    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" containerID="cri-o://f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb" gracePeriod=30
Nov 01 15:04:30.069979 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:30.069949    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:30.465952 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:04:30.465914    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921fae53_caf9_40e6_88f4_6ba8d6996f3c.slice/crio-f03860ec876ae5cfd0d72c1a286a018e1b1cacfdefe19e2fee0c890d5e3cb8a7 WatchSource:0}: Error finding container f03860ec876ae5cfd0d72c1a286a018e1b1cacfdefe19e2fee0c890d5e3cb8a7: Status 404 returned error can't find the container with id f03860ec876ae5cfd0d72c1a286a018e1b1cacfdefe19e2fee0c890d5e3cb8a7
Nov 01 15:04:30.469556 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:30.469503    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"]
Nov 01 15:04:31.025756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:31.025722    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv" event={"ID":"921fae53-caf9-40e6-88f4-6ba8d6996f3c","Type":"ContainerStarted","Data":"b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc"}
Nov 01 15:04:31.025929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:31.025783    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv" event={"ID":"921fae53-caf9-40e6-88f4-6ba8d6996f3c","Type":"ContainerStarted","Data":"f03860ec876ae5cfd0d72c1a286a018e1b1cacfdefe19e2fee0c890d5e3cb8a7"}
Nov 01 15:04:31.025982 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:31.025965    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:31.144001 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:31.143930    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv" podStartSLOduration=2.14391057 podStartE2EDuration="2.14391057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:04:31.142313082 +0000 UTC m=+2811.307798379" watchObservedRunningTime="2026-11-01 15:04:31.14391057 +0000 UTC m=+2811.309395871"
Nov 01 15:04:33.595914 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.595859    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 15:04:33.703882 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.703843    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-envoy\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.703951    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-data\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704030    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-kube-api-access-r9v7c\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-kube-api-access-r9v7c\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704063    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-socket\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704094    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-credential-socket\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704161    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-certs\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704338 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704174    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-data" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.704338 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704239    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-token\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-token\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704338 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704310    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istiod-ca-cert\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704349    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-podinfo\") pod \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\" (UID: \"8e422494-0e14-4a7d-83e6-c29e2030d6bd\") "
Nov 01 15:04:33.704518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704411    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-socket" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.704620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704526    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-credential-socket" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.704735 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704707    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istiod-ca-cert" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:33.704946 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.704920    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.705071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.705006    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-certs" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.705071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.705026    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.705071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.705044    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.705071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.705058    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.706455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.706432    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-kube-api-access-r9v7c" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "kube-api-access-r9v7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:33.706555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.706458    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-token" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:33.706898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.706872    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-envoy" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.707001 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.706874    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-podinfo" pod "8e422494-0e14-4a7d-83e6-c29e2030d6bd" (UID: "8e422494-0e14-4a7d-83e6-c29e2030d6bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:04:33.805893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.805858    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9v7c\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-kube-api-access-r9v7c\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.805893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.805889    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.805893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.805900    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.806120 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.805908    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.806120 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:33.805918    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e422494-0e14-4a7d-83e6-c29e2030d6bd-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:34.049600 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.049559    5423 generic.go:356] "Generic (PLEG): container finished" podID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerID="544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1" exitCode=0
Nov 01 15:04:34.049764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.049634    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"
Nov 01 15:04:34.049764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.049639    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" event={"ID":"8e422494-0e14-4a7d-83e6-c29e2030d6bd","Type":"ContainerDied","Data":"544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1"}
Nov 01 15:04:34.049764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.049694    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7" event={"ID":"8e422494-0e14-4a7d-83e6-c29e2030d6bd","Type":"ContainerDied","Data":"8c3ab47a0dcd4280258380a4589da7a34985bcca84465b967e975aa3c5a95761"}
Nov 01 15:04:34.049764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.049716    5423 scope.go:122] "RemoveContainer" containerID="544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1"
Nov 01 15:04:34.059584 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.059552    5423 scope.go:122] "RemoveContainer" containerID="544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1"
Nov 01 15:04:34.059837 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:04:34.059816    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1\": container with ID starting with 544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1 not found: ID does not exist" containerID="544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1"
Nov 01 15:04:34.059908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.059845    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1"} err="failed to get container status \"544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1\": rpc error: code = NotFound desc = could not find container \"544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1\": container with ID starting with 544f6e8592401b5c5270f98d46073982804f44d833ea3c3932066c0cb3a198a1 not found: ID does not exist"
Nov 01 15:04:34.099687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.099650    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xkd92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:04:34.099803 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.099718    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:04:34.267459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.266811    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"]
Nov 01 15:04:34.267459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:34.266876    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-rrwv7"]
Nov 01 15:04:35.154563 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:04:35.154521    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:04:35.255805 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.255751    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 15:04:35.439588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.439553    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-socket\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.439764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.439644    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-kube-api-access-f8jd4\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-kube-api-access-f8jd4\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.439764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.439667    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-token\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.439764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.439705    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-certs\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.440169 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440137    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-socket" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:35.440255 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440153    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-certs" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:35.440321 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440265    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-podinfo\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.440401 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440335    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-credential-socket\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.440401 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440370    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istiod-ca-cert\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.440609 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440589    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-envoy\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.440794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440755    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-credential-socket" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:35.440925 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440906    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-data\") pod \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\" (UID: \"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5\") "
Nov 01 15:04:35.441020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.440966    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istiod-ca-cert" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:35.441128 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.441046    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-data" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:35.441651 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.441625    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.441749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.441655    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.441749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.441670    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.441749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.441684    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.441749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.441697    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.442782 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.442603    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-token" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:35.444273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.443467    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-kube-api-access-f8jd4" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "kube-api-access-f8jd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:35.444273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.444230    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-podinfo" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:04:35.444273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.444236    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-envoy" pod "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" (UID: "8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:35.542743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.542674    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.542743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.542703    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8jd4\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-kube-api-access-f8jd4\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.542743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.542714    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.542743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.542723    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:35.935459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.934463    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:35.998183 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:35.996313    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" path="/var/lib/kubelet/pods/8e422494-0e14-4a7d-83e6-c29e2030d6bd/volumes"
Nov 01 15:04:36.010941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.010708    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:36.064180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.064146    5423 generic.go:356] "Generic (PLEG): container finished" podID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerID="f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb" exitCode=0
Nov 01 15:04:36.064327 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.064187    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"
Nov 01 15:04:36.064327 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.064222    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" event={"ID":"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5","Type":"ContainerDied","Data":"f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb"}
Nov 01 15:04:36.064327 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.064285    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92" event={"ID":"8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5","Type":"ContainerDied","Data":"1a8e13c4a7c02144709e89b6bb7e57e21952cee91fefe123957266d24e3b8c8e"}
Nov 01 15:04:36.064327 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.064326    5423 scope.go:122] "RemoveContainer" containerID="f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb"
Nov 01 15:04:36.074572 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.074548    5423 scope.go:122] "RemoveContainer" containerID="f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb"
Nov 01 15:04:36.074818 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:04:36.074797    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb\": container with ID starting with f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb not found: ID does not exist" containerID="f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb"
Nov 01 15:04:36.074893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.074830    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb"} err="failed to get container status \"f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb\": rpc error: code = NotFound desc = could not find container \"f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb\": container with ID starting with f09d59dad9d2b6d91eff939fa45234753031ff4392b31677b9a89fa7891905eb not found: ID does not exist"
Nov 01 15:04:36.076961 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.076941    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:04:36.220944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.220859    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"]
Nov 01 15:04:36.221651 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.221631    5423 cpu_manager.go:414] "containerMap: removing container" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy"
Nov 01 15:04:36.221756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.221652    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy"
Nov 01 15:04:36.221756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.221724    5423 cpu_manager.go:414] "containerMap: removing container" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy"
Nov 01 15:04:36.221756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.221733    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy"
Nov 01 15:04:36.221929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.221895    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" containerName="istio-proxy"
Nov 01 15:04:36.221929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.221920    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e422494-0e14-4a7d-83e6-c29e2030d6bd" containerName="istio-proxy"
Nov 01 15:04:36.226645 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.226623    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:36.281380 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.281348    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"]
Nov 01 15:04:36.285666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.285646    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1ac11c-dockercfg-nl2pq\""
Nov 01 15:04:36.341689 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.341661    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1ac11c\""
Nov 01 15:04:36.365375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.365334    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7973261d-8cc2-4f1f-857f-c5d783a54908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh\" (UID: \"7973261d-8cc2-4f1f-857f-c5d783a54908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:36.415857 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.415820    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m27ns\" (UniqueName: \"kubernetes.io/projected/7973261d-8cc2-4f1f-857f-c5d783a54908-kube-api-access-m27ns\") pod \"payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh\" (UID: \"7973261d-8cc2-4f1f-857f-c5d783a54908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:36.416253 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.416218    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"]
Nov 01 15:04:36.422136 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.422110    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xkd92"]
Nov 01 15:04:36.425773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.425749    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"]
Nov 01 15:04:36.437412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.430823    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:36.481612 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.480619    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"]
Nov 01 15:04:36.517578 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.517546    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gvkh\" (UniqueName: \"kubernetes.io/projected/802ca007-2dfb-466e-bb41-a797ad8528b7-kube-api-access-5gvkh\") pod \"payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59\" (UID: \"802ca007-2dfb-466e-bb41-a797ad8528b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:36.517769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.517607    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7973261d-8cc2-4f1f-857f-c5d783a54908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh\" (UID: \"7973261d-8cc2-4f1f-857f-c5d783a54908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:36.517769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.517724    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m27ns\" (UniqueName: \"kubernetes.io/projected/7973261d-8cc2-4f1f-857f-c5d783a54908-kube-api-access-m27ns\") pod \"payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh\" (UID: \"7973261d-8cc2-4f1f-857f-c5d783a54908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:36.517889 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.517814    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802ca007-2dfb-466e-bb41-a797ad8528b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59\" (UID: \"802ca007-2dfb-466e-bb41-a797ad8528b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:36.518434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.518376    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7973261d-8cc2-4f1f-857f-c5d783a54908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh\" (UID: \"7973261d-8cc2-4f1f-857f-c5d783a54908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:36.575689 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.575655    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m27ns\" (UniqueName: \"kubernetes.io/projected/7973261d-8cc2-4f1f-857f-c5d783a54908-kube-api-access-m27ns\") pod \"payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh\" (UID: \"7973261d-8cc2-4f1f-857f-c5d783a54908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:36.619184 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.619150    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802ca007-2dfb-466e-bb41-a797ad8528b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59\" (UID: \"802ca007-2dfb-466e-bb41-a797ad8528b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:36.619357 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.619261    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gvkh\" (UniqueName: \"kubernetes.io/projected/802ca007-2dfb-466e-bb41-a797ad8528b7-kube-api-access-5gvkh\") pod \"payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59\" (UID: \"802ca007-2dfb-466e-bb41-a797ad8528b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:36.619915 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.619890    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802ca007-2dfb-466e-bb41-a797ad8528b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59\" (UID: \"802ca007-2dfb-466e-bb41-a797ad8528b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:36.676974 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.676950    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gvkh\" (UniqueName: \"kubernetes.io/projected/802ca007-2dfb-466e-bb41-a797ad8528b7-kube-api-access-5gvkh\") pod \"payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59\" (UID: \"802ca007-2dfb-466e-bb41-a797ad8528b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:36.777075 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.777005    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:36.855491 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:36.840222    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:37.146576 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:04:37.146542    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802ca007_2dfb_466e_bb41_a797ad8528b7.slice/crio-950def2a9f29a4befc68828545682d826290861790080a1346d7bf25711bbd62 WatchSource:0}: Error finding container 950def2a9f29a4befc68828545682d826290861790080a1346d7bf25711bbd62: Status 404 returned error can't find the container with id 950def2a9f29a4befc68828545682d826290861790080a1346d7bf25711bbd62
Nov 01 15:04:37.149845 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:37.149820    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"]
Nov 01 15:04:37.235306 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:04:37.235275    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7973261d_8cc2_4f1f_857f_c5d783a54908.slice/crio-921a10d3a76c78a79b97ea299d7e159f40e876bf21f4735f0b73380796262c10 WatchSource:0}: Error finding container 921a10d3a76c78a79b97ea299d7e159f40e876bf21f4735f0b73380796262c10: Status 404 returned error can't find the container with id 921a10d3a76c78a79b97ea299d7e159f40e876bf21f4735f0b73380796262c10
Nov 01 15:04:37.238922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:37.238900    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"]
Nov 01 15:04:37.961193 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:37.961163    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5" path="/var/lib/kubelet/pods/8c8cf631-16c6-4ed6-b1ad-5ff4e79ccab5/volumes"
Nov 01 15:04:38.080279 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:38.080244    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" event={"ID":"7973261d-8cc2-4f1f-857f-c5d783a54908","Type":"ContainerStarted","Data":"6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd"}
Nov 01 15:04:38.080457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:38.080301    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" event={"ID":"7973261d-8cc2-4f1f-857f-c5d783a54908","Type":"ContainerStarted","Data":"921a10d3a76c78a79b97ea299d7e159f40e876bf21f4735f0b73380796262c10"}
Nov 01 15:04:38.080549 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:38.080478    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:38.081884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:38.081866    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" event={"ID":"802ca007-2dfb-466e-bb41-a797ad8528b7","Type":"ContainerStarted","Data":"9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac"}
Nov 01 15:04:38.081971 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:38.081898    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" event={"ID":"802ca007-2dfb-466e-bb41-a797ad8528b7","Type":"ContainerStarted","Data":"950def2a9f29a4befc68828545682d826290861790080a1346d7bf25711bbd62"}
Nov 01 15:04:38.082007 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:38.081987    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:38.197158 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:38.197103    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" podStartSLOduration=2.19709049 podStartE2EDuration="2.19709049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:04:38.196554807 +0000 UTC m=+2818.362040104" watchObservedRunningTime="2026-11-01 15:04:38.19709049 +0000 UTC m=+2818.362575787"
Nov 01 15:04:38.320643 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:38.320555    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" podStartSLOduration=2.320436683 podStartE2EDuration="2.320436683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:04:38.312936835 +0000 UTC m=+2818.478422133" watchObservedRunningTime="2026-11-01 15:04:38.320436683 +0000 UTC m=+2818.485921985"
Nov 01 15:04:42.035248 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:42.035217    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:04:49.091773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:49.091745    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:04:49.092153 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:49.091811    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:04:58.886464 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:04:58.886430    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:05:05.150132 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:05:05.150094    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:05:34.940492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:05:34.940453    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:05:35.150961 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:05:35.150926    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:06:05.148713 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:06:05.148673    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:06:09.014445 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:06:09.014413    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:06:35.150234 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:06:35.150199    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:07:02.562243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:07:02.562205    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:07:05.149937 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:07:05.149898    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:07:35.150589 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:07:35.150538    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:07:42.589649 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:07:42.589619    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:07:42.624100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:07:42.624077    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:07:53.401339 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:07:53.401296    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:08:05.149540 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:08:05.149501    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:08:10.763508 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:10.761180    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"]
Nov 01 15:08:15.761830 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:15.761750    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv" podUID="921fae53-caf9-40e6-88f4-6ba8d6996f3c" containerName="payload-pre-processing" containerID="cri-o://b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc" gracePeriod=25
Nov 01 15:08:16.084029 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.083994    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:08:16.250070 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.250032    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921fae53-caf9-40e6-88f4-6ba8d6996f3c-kube-api-access-dd728\" (UniqueName: \"kubernetes.io/projected/921fae53-caf9-40e6-88f4-6ba8d6996f3c-kube-api-access-dd728\") pod \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\" (UID: \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\") "
Nov 01 15:08:16.250280 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.250211    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921fae53-caf9-40e6-88f4-6ba8d6996f3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921fae53-caf9-40e6-88f4-6ba8d6996f3c-plugins-config-volume\") pod \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\" (UID: \"921fae53-caf9-40e6-88f4-6ba8d6996f3c\") "
Nov 01 15:08:16.250715 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.250686    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921fae53-caf9-40e6-88f4-6ba8d6996f3c-plugins-config-volume" pod "921fae53-caf9-40e6-88f4-6ba8d6996f3c" (UID: "921fae53-caf9-40e6-88f4-6ba8d6996f3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:08:16.252248 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.252217    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921fae53-caf9-40e6-88f4-6ba8d6996f3c-kube-api-access-dd728" pod "921fae53-caf9-40e6-88f4-6ba8d6996f3c" (UID: "921fae53-caf9-40e6-88f4-6ba8d6996f3c"). InnerVolumeSpecName "kube-api-access-dd728". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:08:16.352069 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.352026    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd728\" (UniqueName: \"kubernetes.io/projected/921fae53-caf9-40e6-88f4-6ba8d6996f3c-kube-api-access-dd728\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:08:16.352069 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.352058    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921fae53-caf9-40e6-88f4-6ba8d6996f3c-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:08:16.530366 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.530331    5423 generic.go:356] "Generic (PLEG): container finished" podID="921fae53-caf9-40e6-88f4-6ba8d6996f3c" containerID="b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc" exitCode=0
Nov 01 15:08:16.530581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.530402    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"
Nov 01 15:08:16.530581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.530421    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv" event={"ID":"921fae53-caf9-40e6-88f4-6ba8d6996f3c","Type":"ContainerDied","Data":"b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc"}
Nov 01 15:08:16.530581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.530514    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv" event={"ID":"921fae53-caf9-40e6-88f4-6ba8d6996f3c","Type":"ContainerDied","Data":"f03860ec876ae5cfd0d72c1a286a018e1b1cacfdefe19e2fee0c890d5e3cb8a7"}
Nov 01 15:08:16.530581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.530547    5423 scope.go:122] "RemoveContainer" containerID="b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc"
Nov 01 15:08:16.541328 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.541305    5423 scope.go:122] "RemoveContainer" containerID="b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc"
Nov 01 15:08:16.541657 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:08:16.541636    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc\": container with ID starting with b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc not found: ID does not exist" containerID="b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc"
Nov 01 15:08:16.541729 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.541692    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc"} err="failed to get container status \"b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc\": rpc error: code = NotFound desc = could not find container \"b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc\": container with ID starting with b6eb76a6edb0b4d9e31d30d0702dd8efa2da3ddd249025fbf3d17d881caa84bc not found: ID does not exist"
Nov 01 15:08:16.704175 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.704078    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"]
Nov 01 15:08:16.755019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:16.754980    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7794bf-76f64df48c-t5jgv"]
Nov 01 15:08:17.961209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:17.961176    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921fae53-caf9-40e6-88f4-6ba8d6996f3c" path="/var/lib/kubelet/pods/921fae53-caf9-40e6-88f4-6ba8d6996f3c/volumes"
Nov 01 15:08:35.152022 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:08:35.151984    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:08:48.261136 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:08:48.261094    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:09:05.151906 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:09:05.151867    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:09:35.149028 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:09:35.148991    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:09:39.098295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:39.098263    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:09:46.231803 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:46.231773    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"]
Nov 01 15:09:46.232199 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:46.231978    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy" containerID="cri-o://59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0" gracePeriod=30
Nov 01 15:09:51.495416 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.495371    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 15:09:51.500143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500113    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-credential-socket\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500299 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500182    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-envoy\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500299 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500217    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-socket\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500450 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500353    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istiod-ca-cert\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500450 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500403    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-kube-api-access-mvgfv\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-kube-api-access-mvgfv\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500488    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-token\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-token\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500524    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-data\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500621 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500552    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-certs\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500621 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500597    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-podinfo\") pod \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\" (UID: \"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d\") "
Nov 01 15:09:51.500717 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500686    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istiod-ca-cert" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:09:51.500998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500919    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-data" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:51.500998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500773    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-socket" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:51.500998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500968    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-credential-socket" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:51.500998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.500986    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-certs" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:51.501200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.501121    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:51.501200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.501141    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:51.501200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.501156    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:51.501200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.501170    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:51.501200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.501183    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:51.502781 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.502756    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-kube-api-access-mvgfv" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "kube-api-access-mvgfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:09:51.503111 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.503092    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-podinfo" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:09:51.503174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.503139    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-token" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:09:51.503174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.503163    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-envoy" pod "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" (UID: "780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:51.602785 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.602750    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvgfv\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-kube-api-access-mvgfv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:51.602785 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.602784    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:51.602990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.602797    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:51.602990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:51.602810    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:52.207020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.206985    5423 generic.go:356] "Generic (PLEG): container finished" podID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerID="59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0" exitCode=0
Nov 01 15:09:52.207203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.207034    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"
Nov 01 15:09:52.207203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.207066    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" event={"ID":"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d","Type":"ContainerDied","Data":"59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0"}
Nov 01 15:09:52.207203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.207120    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk" event={"ID":"780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d","Type":"ContainerDied","Data":"4e4ffed024557b2cf52dec09f77426300802e42cb5c906341916c88548cd2728"}
Nov 01 15:09:52.207203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.207150    5423 scope.go:122] "RemoveContainer" containerID="59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0"
Nov 01 15:09:52.217619 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.217599    5423 scope.go:122] "RemoveContainer" containerID="59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0"
Nov 01 15:09:52.217893 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:09:52.217875    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0\": container with ID starting with 59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0 not found: ID does not exist" containerID="59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0"
Nov 01 15:09:52.217961 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.217901    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0"} err="failed to get container status \"59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0\": rpc error: code = NotFound desc = could not find container \"59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0\": container with ID starting with 59f008742ca2e5e974fd2795f7e3e282482221864b99c621f7b9901c372c2ce0 not found: ID does not exist"
Nov 01 15:09:52.380810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.380777    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"]
Nov 01 15:09:52.431678 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:52.431650    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-mhwvk"]
Nov 01 15:09:53.966073 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:53.966035    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" path="/var/lib/kubelet/pods/780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d/volumes"
Nov 01 15:09:57.127047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:57.127014    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"]
Nov 01 15:09:57.178431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:57.178374    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"]
Nov 01 15:09:57.178655 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:57.178610    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy" containerID="cri-o://5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9" gracePeriod=30
Nov 01 15:09:57.182036 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:57.182003    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"]
Nov 01 15:09:57.182252 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:57.182210    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy" containerID="cri-o://8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d" gracePeriod=30
Nov 01 15:09:57.241981 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:57.241950    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy" containerID="cri-o://04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3" gracePeriod=30
Nov 01 15:09:59.163818 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:59.163776    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:09:59.164207 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:09:59.163850    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:10:00.198176 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:00.198137    5423 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:10:00.198587 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:00.198191    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:10:02.504397 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.504353    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 15:10:02.525281 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.525258    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 15:10:02.542322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542297    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-kube-api-access-6c24w\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-kube-api-access-6c24w\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.542433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542370    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-envoy\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.542433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542417    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istiod-ca-cert\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.542547 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542447    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-certs\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.542547 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542489    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-data\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.542547 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542537    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-kube-api-access-cz6wl\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-kube-api-access-cz6wl\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.542711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542565    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-envoy\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.542711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542636    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-token\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-token\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.542711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542693    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66320d1e-93ec-46ac-8bfd-623f4605a96a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66320d1e-93ec-46ac-8bfd-623f4605a96a-istiod-ca-cert\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.542887 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542749    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-socket\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.542887 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542805    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-token\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-token\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.542887 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542849    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-podinfo\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.543060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542894    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-data" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.543060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542914    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-data\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.543060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.542971    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-credential-socket\") pod \"66320d1e-93ec-46ac-8bfd-623f4605a96a\" (UID: \"66320d1e-93ec-46ac-8bfd-623f4605a96a\") "
Nov 01 15:10:02.543060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543012    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-certs\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.543280 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543056    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-certs" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.543280 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543070    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-podinfo\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.543280 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543099    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-socket\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.543280 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543132    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istiod-ca-cert" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:10:02.543280 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543142    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-credential-socket\") pod \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\" (UID: \"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120\") "
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543634    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-socket" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543800    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543825    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543841    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543856    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543890    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-credential-socket" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.543908    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66320d1e-93ec-46ac-8bfd-623f4605a96a-istiod-ca-cert" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.544073    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-certs" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.544105    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-socket" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.544432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.544281    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-credential-socket" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.546039 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.546013    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-data" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.547189 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.547168    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-podinfo" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:10:02.547869 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.547845    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-kube-api-access-6c24w" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "kube-api-access-6c24w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:02.548807 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.548773    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-envoy" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.548914 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.548823    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-token" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:02.548914 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.548844    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-envoy" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.549060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.549040    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-podinfo" pod "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" (UID: "2fa138a7-5cd2-4e74-9ca8-26a40b6d1120"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:10:02.549134 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.549114    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-token" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:02.549340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.549326    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-kube-api-access-cz6wl" pod "66320d1e-93ec-46ac-8bfd-623f4605a96a" (UID: "66320d1e-93ec-46ac-8bfd-623f4605a96a"). InnerVolumeSpecName "kube-api-access-cz6wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:02.575080 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.575060    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 15:10:02.645107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645074    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-token\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-token\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645111    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-certs\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645293 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645156    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istiod-ca-cert\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645293 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645185    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-credential-socket\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645293 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645226    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-socket\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645293 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645275    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-envoy\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645519 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645298    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-kube-api-access-fsbph\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-kube-api-access-fsbph\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645519 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645339    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-data\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645519 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645366    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-podinfo\") pod \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\" (UID: \"fecd34a5-f6a3-4e5d-91c2-02663eb01d72\") "
Nov 01 15:10:02.645681 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645511    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-certs" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.645681 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645530    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-credential-socket" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.645681 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645666    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-socket" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.645856 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645686    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istiod-ca-cert" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:10:02.645856 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645847    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-data" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.645981 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645958    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66320d1e-93ec-46ac-8bfd-623f4605a96a-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.645981 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645982    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.645996    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646009    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646021    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646034    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646046    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646059    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646073    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646086    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646099    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646112    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646125    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646140    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c24w\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-kube-api-access-6c24w\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646156    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646170    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646682 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646184    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz6wl\" (UniqueName: \"kubernetes.io/projected/66320d1e-93ec-46ac-8bfd-623f4605a96a-kube-api-access-cz6wl\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646682 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646198    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66320d1e-93ec-46ac-8bfd-623f4605a96a-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.646682 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.646213    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.647577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.647551    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-kube-api-access-fsbph" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "kube-api-access-fsbph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:02.647675 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.647599    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-podinfo" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:10:02.647852 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.647829    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-envoy" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:02.647896 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.647876    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-token" pod "fecd34a5-f6a3-4e5d-91c2-02663eb01d72" (UID: "fecd34a5-f6a3-4e5d-91c2-02663eb01d72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:02.746998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.746913    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.746998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.746941    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbph\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-kube-api-access-fsbph\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.746998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.746952    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:02.746998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:02.746963    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecd34a5-f6a3-4e5d-91c2-02663eb01d72-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:03.278509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.278470    5423 generic.go:356] "Generic (PLEG): container finished" podID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerID="04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3" exitCode=0
Nov 01 15:10:03.278680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.278541    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"
Nov 01 15:10:03.278680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.278552    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" event={"ID":"fecd34a5-f6a3-4e5d-91c2-02663eb01d72","Type":"ContainerDied","Data":"04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3"}
Nov 01 15:10:03.278680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.278623    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk" event={"ID":"fecd34a5-f6a3-4e5d-91c2-02663eb01d72","Type":"ContainerDied","Data":"148630c6f31be484ad900451f3ebadbb0ffd177cd1d493861e232c796bfdd5a2"}
Nov 01 15:10:03.278680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.278652    5423 scope.go:122] "RemoveContainer" containerID="04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3"
Nov 01 15:10:03.280040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.280015    5423 generic.go:356] "Generic (PLEG): container finished" podID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerID="8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d" exitCode=0
Nov 01 15:10:03.280123 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.280071    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"
Nov 01 15:10:03.280172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.280104    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" event={"ID":"66320d1e-93ec-46ac-8bfd-623f4605a96a","Type":"ContainerDied","Data":"8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d"}
Nov 01 15:10:03.280222 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.280171    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67" event={"ID":"66320d1e-93ec-46ac-8bfd-623f4605a96a","Type":"ContainerDied","Data":"ccfffa49d5ac0bb3bdcfb48d2a097ed4711a8060b8438c6588f9bfc4ec631e17"}
Nov 01 15:10:03.281786 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.281761    5423 generic.go:356] "Generic (PLEG): container finished" podID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerID="5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9" exitCode=0
Nov 01 15:10:03.281897 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.281828    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"
Nov 01 15:10:03.281897 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.281851    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" event={"ID":"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120","Type":"ContainerDied","Data":"5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9"}
Nov 01 15:10:03.281989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.281910    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9" event={"ID":"2fa138a7-5cd2-4e74-9ca8-26a40b6d1120","Type":"ContainerDied","Data":"f050be95e236fc3a580bcd80ed593543462ff486b4b668876f209cc90a7d5c92"}
Nov 01 15:10:03.290123 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.290100    5423 scope.go:122] "RemoveContainer" containerID="04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3"
Nov 01 15:10:03.290461 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:03.290435    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3\": container with ID starting with 04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3 not found: ID does not exist" containerID="04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3"
Nov 01 15:10:03.290867 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.290471    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3"} err="failed to get container status \"04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3\": rpc error: code = NotFound desc = could not find container \"04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3\": container with ID starting with 04ca2e18ee35dc4e4b69c62182bbc46c412a9566bbb044166518c97d53116dd3 not found: ID does not exist"
Nov 01 15:10:03.290867 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.290491    5423 scope.go:122] "RemoveContainer" containerID="8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d"
Nov 01 15:10:03.300008 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.299987    5423 scope.go:122] "RemoveContainer" containerID="8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d"
Nov 01 15:10:03.300242 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:03.300224    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d\": container with ID starting with 8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d not found: ID does not exist" containerID="8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d"
Nov 01 15:10:03.300305 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.300249    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d"} err="failed to get container status \"8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d\": rpc error: code = NotFound desc = could not find container \"8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d\": container with ID starting with 8d472972c4e8a57f4c4bd1273b96381d20e711776ecc49deaa0fd2bfbeb1d22d not found: ID does not exist"
Nov 01 15:10:03.300305 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.300269    5423 scope.go:122] "RemoveContainer" containerID="5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9"
Nov 01 15:10:03.308447 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.308431    5423 scope.go:122] "RemoveContainer" containerID="5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9"
Nov 01 15:10:03.308712 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:03.308693    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9\": container with ID starting with 5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9 not found: ID does not exist" containerID="5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9"
Nov 01 15:10:03.308766 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.308719    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9"} err="failed to get container status \"5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9\": rpc error: code = NotFound desc = could not find container \"5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9\": container with ID starting with 5ef300847e10b2b3a5b933a63af7a88bb4190bf0b375901d4733cd7695331bf9 not found: ID does not exist"
Nov 01 15:10:03.447624 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.447585    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"]
Nov 01 15:10:03.498483 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.498417    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-zzwvk"]
Nov 01 15:10:03.609806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.609773    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"]
Nov 01 15:10:03.613950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.613926    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m8q67"]
Nov 01 15:10:03.728666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.728615    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"]
Nov 01 15:10:03.736302 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.736275    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kkfd9"]
Nov 01 15:10:03.961243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.961208    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" path="/var/lib/kubelet/pods/2fa138a7-5cd2-4e74-9ca8-26a40b6d1120/volumes"
Nov 01 15:10:03.961772 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.961756    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" path="/var/lib/kubelet/pods/66320d1e-93ec-46ac-8bfd-623f4605a96a/volumes"
Nov 01 15:10:03.962185 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:03.962172    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" path="/var/lib/kubelet/pods/fecd34a5-f6a3-4e5d-91c2-02663eb01d72/volumes"
Nov 01 15:10:05.150912 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:05.150876    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:10:05.225289 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225260    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-0968fe"]
Nov 01 15:10:05.225818 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225804    5423 cpu_manager.go:414] "containerMap: removing container" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy"
Nov 01 15:10:05.225861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225820    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy"
Nov 01 15:10:05.225861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225832    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy"
Nov 01 15:10:05.225861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225838    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy"
Nov 01 15:10:05.225861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225851    5423 cpu_manager.go:414] "containerMap: removing container" podUID="921fae53-caf9-40e6-88f4-6ba8d6996f3c" containerName="payload-pre-processing"
Nov 01 15:10:05.225861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225858    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921fae53-caf9-40e6-88f4-6ba8d6996f3c" containerName="payload-pre-processing"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225867    5423 cpu_manager.go:414] "containerMap: removing container" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225872    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225881    5423 cpu_manager.go:414] "containerMap: removing container" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225887    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225969    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa138a7-5cd2-4e74-9ca8-26a40b6d1120" containerName="istio-proxy"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225976    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecd34a5-f6a3-4e5d-91c2-02663eb01d72" containerName="istio-proxy"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225983    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="921fae53-caf9-40e6-88f4-6ba8d6996f3c" containerName="payload-pre-processing"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.225990    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="780b6486-e5ca-4a6c-a6b6-4b7d5d2def5d" containerName="istio-proxy"
Nov 01 15:10:05.226060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.226017    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="66320d1e-93ec-46ac-8bfd-623f4605a96a" containerName="istio-proxy"
Nov 01 15:10:05.231255 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.231235    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:05.283173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.283146    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-0968fe"]
Nov 01 15:10:05.285023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.284995    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzxv\" (UniqueName: \"kubernetes.io/projected/16eb3c34-3143-4f8d-81ea-7685286fea34-kube-api-access-7dzxv\") pod \"test-curl-7384-0968fe\" (UID: \"16eb3c34-3143-4f8d-81ea-7685286fea34\") " pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:05.331420 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.331372    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jngw\""
Nov 01 15:10:05.386819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.386791    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzxv\" (UniqueName: \"kubernetes.io/projected/16eb3c34-3143-4f8d-81ea-7685286fea34-kube-api-access-7dzxv\") pod \"test-curl-7384-0968fe\" (UID: \"16eb3c34-3143-4f8d-81ea-7685286fea34\") " pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:05.445345 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.445290    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzxv\" (UniqueName: \"kubernetes.io/projected/16eb3c34-3143-4f8d-81ea-7685286fea34-kube-api-access-7dzxv\") pod \"test-curl-7384-0968fe\" (UID: \"16eb3c34-3143-4f8d-81ea-7685286fea34\") " pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:05.543818 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.543795    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:05.923963 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:10:05.923927    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16eb3c34_3143_4f8d_81ea_7685286fea34.slice/crio-c538fe93e56c043c43aa037a39620a68b74c4da1fabfb9196da9ff54c941df33 WatchSource:0}: Error finding container c538fe93e56c043c43aa037a39620a68b74c4da1fabfb9196da9ff54c941df33: Status 404 returned error can't find the container with id c538fe93e56c043c43aa037a39620a68b74c4da1fabfb9196da9ff54c941df33
Nov 01 15:10:05.927121 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:05.927097    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-0968fe"]
Nov 01 15:10:06.303545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:06.303457    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-0968fe" event={"ID":"16eb3c34-3143-4f8d-81ea-7685286fea34","Type":"ContainerStarted","Data":"f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6"}
Nov 01 15:10:06.303545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:06.303508    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-0968fe" event={"ID":"16eb3c34-3143-4f8d-81ea-7685286fea34","Type":"ContainerStarted","Data":"c538fe93e56c043c43aa037a39620a68b74c4da1fabfb9196da9ff54c941df33"}
Nov 01 15:10:06.417900 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:06.417848    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-0968fe" podStartSLOduration=1.417836064 podStartE2EDuration="1.417836064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:06.417333142 +0000 UTC m=+3146.582818439" watchObservedRunningTime="2026-11-01 15:10:06.417836064 +0000 UTC m=+3146.583321361"
Nov 01 15:10:07.993214 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:07.993180    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-0968fe"]
Nov 01 15:10:08.000821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:08.000793    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-0968fe"]
Nov 01 15:10:08.314880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:08.314848    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-0968fe" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" containerName="test-curl-7384-0968fe" containerID="cri-o://f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6" gracePeriod=2
Nov 01 15:10:08.367487 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:08.367451    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-0968fe\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:09.331242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:09.331206    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"]
Nov 01 15:10:09.531893 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:09.531857    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-0968fe\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:10.019720 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:10.019675    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-0968fe\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:10.317538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:10.317501    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"]
Nov 01 15:10:10.551875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:10.551849    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:10.653547 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:10.653470    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16eb3c34-3143-4f8d-81ea-7685286fea34-kube-api-access-7dzxv\" (UniqueName: \"kubernetes.io/projected/16eb3c34-3143-4f8d-81ea-7685286fea34-kube-api-access-7dzxv\") pod \"16eb3c34-3143-4f8d-81ea-7685286fea34\" (UID: \"16eb3c34-3143-4f8d-81ea-7685286fea34\") "
Nov 01 15:10:10.655568 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:10.655541    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16eb3c34-3143-4f8d-81ea-7685286fea34-kube-api-access-7dzxv" pod "16eb3c34-3143-4f8d-81ea-7685286fea34" (UID: "16eb3c34-3143-4f8d-81ea-7685286fea34"). InnerVolumeSpecName "kube-api-access-7dzxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:10.755588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:10.755556    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzxv\" (UniqueName: \"kubernetes.io/projected/16eb3c34-3143-4f8d-81ea-7685286fea34-kube-api-access-7dzxv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:11.342140 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:11.342102    5423 generic.go:356] "Generic (PLEG): container finished" podID="16eb3c34-3143-4f8d-81ea-7685286fea34" containerID="f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6" exitCode=137
Nov 01 15:10:11.342140 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:11.342140    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:11.342360 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:11.342192    5423 scope.go:122] "RemoveContainer" containerID="f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6"
Nov 01 15:10:11.357330 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:11.357306    5423 scope.go:122] "RemoveContainer" containerID="f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6"
Nov 01 15:10:11.357625 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:11.357604    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6\": container with ID starting with f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6 not found: ID does not exist" containerID="f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6"
Nov 01 15:10:11.357694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:11.357634    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6"} err="failed to get container status \"f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6\": rpc error: code = NotFound desc = could not find container \"f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6\": container with ID starting with f944d570f1f7a2d6c8b37bfa143e97bb89118102fea71a4ff203828e5cd145c6 not found: ID does not exist"
Nov 01 15:10:11.408998 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:11.408939    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-0968fe\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" pod="opendatahub/test-curl-7384-0968fe"
Nov 01 15:10:11.960360 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:11.960329    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" path="/var/lib/kubelet/pods/16eb3c34-3143-4f8d-81ea-7685286fea34/volumes"
Nov 01 15:10:19.942382 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:19.942309    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"]
Nov 01 15:10:19.942883 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:19.942869    5423 cpu_manager.go:414] "containerMap: removing container" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" containerName="test-curl-7384-0968fe"
Nov 01 15:10:19.942931 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:19.942884    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" containerName="test-curl-7384-0968fe"
Nov 01 15:10:19.943005 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:19.942995    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="16eb3c34-3143-4f8d-81ea-7685286fea34" containerName="test-curl-7384-0968fe"
Nov 01 15:10:19.948453 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:19.948433    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.021894 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.021858    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ea1620-openshift-default-dockercfg-gfxzv\""
Nov 01 15:10:20.055900 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.055874    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"]
Nov 01 15:10:20.069543 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069515    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtnp\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-kube-api-access-pwtnp\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.069677 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069557    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-socket\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.069677 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069583    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b165a105-0c29-4a2a-800b-76325486d222-istio-podinfo\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.069758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069681    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b165a105-0c29-4a2a-800b-76325486d222-istiod-ca-cert\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.069758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069723    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-envoy\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.069836 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069778    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-certs\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.069836 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069799    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-istio-token\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.069923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069900    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-data\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.069993 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.069980    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-credential-socket\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.170939 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.170905    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b165a105-0c29-4a2a-800b-76325486d222-istiod-ca-cert\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.170961    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-envoy\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171006    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-certs\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171031    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-istio-token\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171068    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-data\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171324 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171130    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-credential-socket\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171324 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171224    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtnp\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-kube-api-access-pwtnp\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171324 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171263    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-socket\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171324 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171291    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b165a105-0c29-4a2a-800b-76325486d222-istio-podinfo\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171619 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171454    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-certs\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171619 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171524    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-data\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171786 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171753    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b165a105-0c29-4a2a-800b-76325486d222-istiod-ca-cert\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171916 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171787    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-socket\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.171916 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.171863    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-credential-socket\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.173446 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.173426    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-envoy\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.173853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.173831    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b165a105-0c29-4a2a-800b-76325486d222-istio-podinfo\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.190891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.190865    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"]
Nov 01 15:10:20.195626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.195580    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:20.240788 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.240747    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-istio-token\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.241026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.241006    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtnp\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-kube-api-access-pwtnp\") pod \"e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") " pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.250486 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.250459    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2cf1ee\""
Nov 01 15:10:20.250662 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.250525    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2cf1ee-dockercfg-64np5\""
Nov 01 15:10:20.254294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.254273    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"]
Nov 01 15:10:20.263646 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.263623    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:20.276530 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.276465    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vxm\" (UniqueName: \"kubernetes.io/projected/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-kube-api-access-r4vxm\") pod \"payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn\" (UID: \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:20.276666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.276641    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn\" (UID: \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:20.398409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.398167    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vxm\" (UniqueName: \"kubernetes.io/projected/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-kube-api-access-r4vxm\") pod \"payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn\" (UID: \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:20.398409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.398326    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn\" (UID: \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:20.406433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.400064    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn\" (UID: \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:20.466712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.466650    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vxm\" (UniqueName: \"kubernetes.io/projected/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-kube-api-access-r4vxm\") pod \"payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn\" (UID: \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:20.509823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.509796    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:20.794306 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:10:20.794272    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb165a105_0c29_4a2a_800b_76325486d222.slice/crio-715841df8b98a026475f4a77de0467a82b0b89e8b2c280b36e9627435dd9f713 WatchSource:0}: Error finding container 715841df8b98a026475f4a77de0467a82b0b89e8b2c280b36e9627435dd9f713: Status 404 returned error can't find the container with id 715841df8b98a026475f4a77de0467a82b0b89e8b2c280b36e9627435dd9f713
Nov 01 15:10:20.797445 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.797412    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:10:20.797535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.797483    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:10:20.797535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.797513    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:10:20.797752 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.797731    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"]
Nov 01 15:10:20.897692 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:10:20.897658    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d92a10_1b66_40f0_9744_4e4ea0dc9ead.slice/crio-8423411678f51051dc069fa128c62ea51aa02916a69a115688b3ede203e4a404 WatchSource:0}: Error finding container 8423411678f51051dc069fa128c62ea51aa02916a69a115688b3ede203e4a404: Status 404 returned error can't find the container with id 8423411678f51051dc069fa128c62ea51aa02916a69a115688b3ede203e4a404
Nov 01 15:10:20.901441 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:20.901367    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"]
Nov 01 15:10:21.296811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:21.296766    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:10:21.486035 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:21.485997    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn" event={"ID":"78d92a10-1b66-40f0-9744-4e4ea0dc9ead","Type":"ContainerStarted","Data":"62c2d60c15f1e6600209620f8805ff49df53d2b33bca7def6a309d1e4cae940b"}
Nov 01 15:10:21.491663 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:21.491553    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn" event={"ID":"78d92a10-1b66-40f0-9744-4e4ea0dc9ead","Type":"ContainerStarted","Data":"8423411678f51051dc069fa128c62ea51aa02916a69a115688b3ede203e4a404"}
Nov 01 15:10:21.491933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:21.491914    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:21.496956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:21.496926    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" event={"ID":"b165a105-0c29-4a2a-800b-76325486d222","Type":"ContainerStarted","Data":"6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907"}
Nov 01 15:10:21.497143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:21.497128    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" event={"ID":"b165a105-0c29-4a2a-800b-76325486d222","Type":"ContainerStarted","Data":"715841df8b98a026475f4a77de0467a82b0b89e8b2c280b36e9627435dd9f713"}
Nov 01 15:10:21.609062 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:21.609003    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn" podStartSLOduration=1.608987039 podStartE2EDuration="1.608987039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:21.607009909 +0000 UTC m=+3161.772495208" watchObservedRunningTime="2026-11-01 15:10:21.608987039 +0000 UTC m=+3161.774472331"
Nov 01 15:10:21.729751 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:21.729702    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podStartSLOduration=2.729687307 podStartE2EDuration="2.729687307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:21.726624775 +0000 UTC m=+3161.892110073" watchObservedRunningTime="2026-11-01 15:10:21.729687307 +0000 UTC m=+3161.895172605"
Nov 01 15:10:22.264109 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:22.264075    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:22.270344 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:22.270318    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:22.503437 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:22.503408    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:22.504447 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:22.504425    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:10:24.352127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.352082    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq" podUID="ae153428-61eb-4192-842e-74cb00137fa6" containerName="main" containerID="cri-o://3cd9b7e4df4d1d7a5d1504eb10dfc2d485556db32f10f326ad2058a89e798e40" gracePeriod=45
Nov 01 15:10:24.517031 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.517000    5423 generic.go:356] "Generic (PLEG): container finished" podID="ae153428-61eb-4192-842e-74cb00137fa6" containerID="3cd9b7e4df4d1d7a5d1504eb10dfc2d485556db32f10f326ad2058a89e798e40" exitCode=0
Nov 01 15:10:24.517197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.517081    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq" event={"ID":"ae153428-61eb-4192-842e-74cb00137fa6","Type":"ContainerDied","Data":"3cd9b7e4df4d1d7a5d1504eb10dfc2d485556db32f10f326ad2058a89e798e40"}
Nov 01 15:10:24.621980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.621955    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:10:24.756247 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756216    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae153428-61eb-4192-842e-74cb00137fa6-kube-api-access-vmqlq\" (UniqueName: \"kubernetes.io/projected/ae153428-61eb-4192-842e-74cb00137fa6-kube-api-access-vmqlq\") pod \"ae153428-61eb-4192-842e-74cb00137fa6\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") "
Nov 01 15:10:24.756444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756269    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-model-cache\") pod \"ae153428-61eb-4192-842e-74cb00137fa6\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") "
Nov 01 15:10:24.756444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756350    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-tmp-dir\") pod \"ae153428-61eb-4192-842e-74cb00137fa6\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") "
Nov 01 15:10:24.756579 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756512    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae153428-61eb-4192-842e-74cb00137fa6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae153428-61eb-4192-842e-74cb00137fa6-tls-certs\") pod \"ae153428-61eb-4192-842e-74cb00137fa6\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") "
Nov 01 15:10:24.756579 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756554    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-home\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-home\") pod \"ae153428-61eb-4192-842e-74cb00137fa6\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") "
Nov 01 15:10:24.756579 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756558    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-model-cache" pod "ae153428-61eb-4192-842e-74cb00137fa6" (UID: "ae153428-61eb-4192-842e-74cb00137fa6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:24.756755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756598    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-tmp-dir" pod "ae153428-61eb-4192-842e-74cb00137fa6" (UID: "ae153428-61eb-4192-842e-74cb00137fa6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:24.756755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756625    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-dshm\") pod \"ae153428-61eb-4192-842e-74cb00137fa6\" (UID: \"ae153428-61eb-4192-842e-74cb00137fa6\") "
Nov 01 15:10:24.756853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.756835    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-home" pod "ae153428-61eb-4192-842e-74cb00137fa6" (UID: "ae153428-61eb-4192-842e-74cb00137fa6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:24.757120 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.757103    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:24.757216 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.757125    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:24.757216 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.757141    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:24.758778 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.758749    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-dshm" pod "ae153428-61eb-4192-842e-74cb00137fa6" (UID: "ae153428-61eb-4192-842e-74cb00137fa6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:24.758887 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.758790    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae153428-61eb-4192-842e-74cb00137fa6-kube-api-access-vmqlq" pod "ae153428-61eb-4192-842e-74cb00137fa6" (UID: "ae153428-61eb-4192-842e-74cb00137fa6"). InnerVolumeSpecName "kube-api-access-vmqlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:24.758951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.758913    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae153428-61eb-4192-842e-74cb00137fa6-tls-certs" pod "ae153428-61eb-4192-842e-74cb00137fa6" (UID: "ae153428-61eb-4192-842e-74cb00137fa6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:10:24.858809 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.858729    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae153428-61eb-4192-842e-74cb00137fa6-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:24.858809 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.858757    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae153428-61eb-4192-842e-74cb00137fa6-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:24.858809 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:24.858767    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmqlq\" (UniqueName: \"kubernetes.io/projected/ae153428-61eb-4192-842e-74cb00137fa6-kube-api-access-vmqlq\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:25.342079 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.342035    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c" podUID="81550cdb-ffc0-4721-8d0f-fa875acb002d" containerName="main" containerID="cri-o://e94ed62b18d8d37c666c638580ea89b0e2974d935566896a097feb1594a41878" gracePeriod=45
Nov 01 15:10:25.527922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.527891    5423 generic.go:356] "Generic (PLEG): container finished" podID="81550cdb-ffc0-4721-8d0f-fa875acb002d" containerID="e94ed62b18d8d37c666c638580ea89b0e2974d935566896a097feb1594a41878" exitCode=0
Nov 01 15:10:25.528313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.527969    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c" event={"ID":"81550cdb-ffc0-4721-8d0f-fa875acb002d","Type":"ContainerDied","Data":"e94ed62b18d8d37c666c638580ea89b0e2974d935566896a097feb1594a41878"}
Nov 01 15:10:25.529810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.529787    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"
Nov 01 15:10:25.529810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.529801    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq" event={"ID":"ae153428-61eb-4192-842e-74cb00137fa6","Type":"ContainerDied","Data":"62478eb0fdb4a1fe9121179cb6c33a45befb91c8d59bedf8ede21f691ad92aeb"}
Nov 01 15:10:25.530025 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.529858    5423 scope.go:122] "RemoveContainer" containerID="3cd9b7e4df4d1d7a5d1504eb10dfc2d485556db32f10f326ad2058a89e798e40"
Nov 01 15:10:25.604638 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.604587    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:10:25.701998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.701971    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"]
Nov 01 15:10:25.752412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.752369    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/auth-test-model-ca0f9e-kserve-7856cb6f7c-xzrqq"]
Nov 01 15:10:25.767981 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.767955    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81550cdb-ffc0-4721-8d0f-fa875acb002d-kube-api-access-ggt2l\" (UniqueName: \"kubernetes.io/projected/81550cdb-ffc0-4721-8d0f-fa875acb002d-kube-api-access-ggt2l\") pod \"81550cdb-ffc0-4721-8d0f-fa875acb002d\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") "
Nov 01 15:10:25.768111 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.767992    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81550cdb-ffc0-4721-8d0f-fa875acb002d-tls-certs\" (UniqueName: \"kubernetes.io/secret/81550cdb-ffc0-4721-8d0f-fa875acb002d-tls-certs\") pod \"81550cdb-ffc0-4721-8d0f-fa875acb002d\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") "
Nov 01 15:10:25.768111 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768023    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-tmp-dir\") pod \"81550cdb-ffc0-4721-8d0f-fa875acb002d\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") "
Nov 01 15:10:25.768243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768129    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-dshm\") pod \"81550cdb-ffc0-4721-8d0f-fa875acb002d\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") "
Nov 01 15:10:25.768243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768146    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-model-cache\") pod \"81550cdb-ffc0-4721-8d0f-fa875acb002d\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") "
Nov 01 15:10:25.768243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768189    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-home\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-home\") pod \"81550cdb-ffc0-4721-8d0f-fa875acb002d\" (UID: \"81550cdb-ffc0-4721-8d0f-fa875acb002d\") "
Nov 01 15:10:25.768453 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768324    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-tmp-dir" pod "81550cdb-ffc0-4721-8d0f-fa875acb002d" (UID: "81550cdb-ffc0-4721-8d0f-fa875acb002d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:25.768513 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768466    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-model-cache" pod "81550cdb-ffc0-4721-8d0f-fa875acb002d" (UID: "81550cdb-ffc0-4721-8d0f-fa875acb002d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:25.768562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768546    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-home" pod "81550cdb-ffc0-4721-8d0f-fa875acb002d" (UID: "81550cdb-ffc0-4721-8d0f-fa875acb002d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:25.768697 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768678    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:25.768736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768702    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:25.768736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.768712    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:25.770334 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.770314    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81550cdb-ffc0-4721-8d0f-fa875acb002d-tls-certs" pod "81550cdb-ffc0-4721-8d0f-fa875acb002d" (UID: "81550cdb-ffc0-4721-8d0f-fa875acb002d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:10:25.770571 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.770554    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-dshm" pod "81550cdb-ffc0-4721-8d0f-fa875acb002d" (UID: "81550cdb-ffc0-4721-8d0f-fa875acb002d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:25.770630 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.770612    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81550cdb-ffc0-4721-8d0f-fa875acb002d-kube-api-access-ggt2l" pod "81550cdb-ffc0-4721-8d0f-fa875acb002d" (UID: "81550cdb-ffc0-4721-8d0f-fa875acb002d"). InnerVolumeSpecName "kube-api-access-ggt2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:25.870337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.870261    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggt2l\" (UniqueName: \"kubernetes.io/projected/81550cdb-ffc0-4721-8d0f-fa875acb002d-kube-api-access-ggt2l\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:25.870337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.870290    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81550cdb-ffc0-4721-8d0f-fa875acb002d-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:25.870337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.870302    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81550cdb-ffc0-4721-8d0f-fa875acb002d-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:25.960083 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:25.960049    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae153428-61eb-4192-842e-74cb00137fa6" path="/var/lib/kubelet/pods/ae153428-61eb-4192-842e-74cb00137fa6/volumes"
Nov 01 15:10:26.536790 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:26.536758    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"
Nov 01 15:10:26.537277 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:26.536788    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c" event={"ID":"81550cdb-ffc0-4721-8d0f-fa875acb002d","Type":"ContainerDied","Data":"d49cdec5740c96f12c02e487dd2be1730a644ca3206853f4f9838ea0c0185aaa"}
Nov 01 15:10:26.537277 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:26.536844    5423 scope.go:122] "RemoveContainer" containerID="e94ed62b18d8d37c666c638580ea89b0e2974d935566896a097feb1594a41878"
Nov 01 15:10:26.703729 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:26.703692    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"]
Nov 01 15:10:26.754649 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:26.754614    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-97c87c/auth-test-model-97c87c-kserve-54775d54cc-84k7c"]
Nov 01 15:10:27.963363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:27.963334    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81550cdb-ffc0-4721-8d0f-fa875acb002d" path="/var/lib/kubelet/pods/81550cdb-ffc0-4721-8d0f-fa875acb002d/volumes"
Nov 01 15:10:29.104039 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.104003    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"]
Nov 01 15:10:29.104567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.104553    5423 cpu_manager.go:414] "containerMap: removing container" podUID="81550cdb-ffc0-4721-8d0f-fa875acb002d" containerName="main"
Nov 01 15:10:29.104627 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.104567    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81550cdb-ffc0-4721-8d0f-fa875acb002d" containerName="main"
Nov 01 15:10:29.104627 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.104605    5423 cpu_manager.go:414] "containerMap: removing container" podUID="ae153428-61eb-4192-842e-74cb00137fa6" containerName="main"
Nov 01 15:10:29.104627 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.104610    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae153428-61eb-4192-842e-74cb00137fa6" containerName="main"
Nov 01 15:10:29.104725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.104690    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae153428-61eb-4192-842e-74cb00137fa6" containerName="main"
Nov 01 15:10:29.104725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.104700    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="81550cdb-ffc0-4721-8d0f-fa875acb002d" containerName="main"
Nov 01 15:10:29.109747 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.109722    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:29.178406 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.176505    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"]
Nov 01 15:10:29.219008 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.218966    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ea1620\""
Nov 01 15:10:29.220565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.219564    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ea1620-dockercfg-gtdnf\""
Nov 01 15:10:29.222086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.222058    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d5c107-3074-456e-b8a3-c7f7db06291e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7\" (UID: \"d0d5c107-3074-456e-b8a3-c7f7db06291e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:29.222492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.222459    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccs5x\" (UniqueName: \"kubernetes.io/projected/d0d5c107-3074-456e-b8a3-c7f7db06291e-kube-api-access-ccs5x\") pod \"payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7\" (UID: \"d0d5c107-3074-456e-b8a3-c7f7db06291e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:29.273799 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.273763    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"]
Nov 01 15:10:29.280250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.280226    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:29.311816 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.311784    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"]
Nov 01 15:10:29.323601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.323568    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccs5x\" (UniqueName: \"kubernetes.io/projected/d0d5c107-3074-456e-b8a3-c7f7db06291e-kube-api-access-ccs5x\") pod \"payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7\" (UID: \"d0d5c107-3074-456e-b8a3-c7f7db06291e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:29.323749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.323694    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d5c107-3074-456e-b8a3-c7f7db06291e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7\" (UID: \"d0d5c107-3074-456e-b8a3-c7f7db06291e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:29.324425 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.324374    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d5c107-3074-456e-b8a3-c7f7db06291e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7\" (UID: \"d0d5c107-3074-456e-b8a3-c7f7db06291e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:29.382269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.382194    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccs5x\" (UniqueName: \"kubernetes.io/projected/d0d5c107-3074-456e-b8a3-c7f7db06291e-kube-api-access-ccs5x\") pod \"payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7\" (UID: \"d0d5c107-3074-456e-b8a3-c7f7db06291e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:29.423530 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.423493    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:29.424632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.424607    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/039004c4-1357-4304-850b-c0b807ab2da3-kube-api-access-ccd7w\") pod \"payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv\" (UID: \"039004c4-1357-4304-850b-c0b807ab2da3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:29.424775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.424753    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039004c4-1357-4304-850b-c0b807ab2da3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv\" (UID: \"039004c4-1357-4304-850b-c0b807ab2da3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:29.526163 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.526126    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039004c4-1357-4304-850b-c0b807ab2da3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv\" (UID: \"039004c4-1357-4304-850b-c0b807ab2da3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:29.526325 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.526306    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/039004c4-1357-4304-850b-c0b807ab2da3-kube-api-access-ccd7w\") pod \"payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv\" (UID: \"039004c4-1357-4304-850b-c0b807ab2da3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:29.526960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.526935    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039004c4-1357-4304-850b-c0b807ab2da3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv\" (UID: \"039004c4-1357-4304-850b-c0b807ab2da3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:29.586249 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.586224    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/039004c4-1357-4304-850b-c0b807ab2da3-kube-api-access-ccd7w\") pod \"payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv\" (UID: \"039004c4-1357-4304-850b-c0b807ab2da3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:29.595537 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.595510    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:29.794051 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:10:29.794010    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d5c107_3074_456e_b8a3_c7f7db06291e.slice/crio-42c3cb3137eafeca40919640b75e86b640f671d425884d12928e8b8942b9e626 WatchSource:0}: Error finding container 42c3cb3137eafeca40919640b75e86b640f671d425884d12928e8b8942b9e626: Status 404 returned error can't find the container with id 42c3cb3137eafeca40919640b75e86b640f671d425884d12928e8b8942b9e626
Nov 01 15:10:29.797554 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.797529    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"]
Nov 01 15:10:29.967797 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:10:29.967768    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039004c4_1357_4304_850b_c0b807ab2da3.slice/crio-8fb91f44d72aeca311f6a18a1dd434d3d0ba91dbcf872f33ea9dba90d8a3cf1f WatchSource:0}: Error finding container 8fb91f44d72aeca311f6a18a1dd434d3d0ba91dbcf872f33ea9dba90d8a3cf1f: Status 404 returned error can't find the container with id 8fb91f44d72aeca311f6a18a1dd434d3d0ba91dbcf872f33ea9dba90d8a3cf1f
Nov 01 15:10:29.970884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:29.970859    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"]
Nov 01 15:10:30.565459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:30.565420    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv" event={"ID":"039004c4-1357-4304-850b-c0b807ab2da3","Type":"ContainerStarted","Data":"477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995"}
Nov 01 15:10:30.565888 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:30.565485    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv" event={"ID":"039004c4-1357-4304-850b-c0b807ab2da3","Type":"ContainerStarted","Data":"8fb91f44d72aeca311f6a18a1dd434d3d0ba91dbcf872f33ea9dba90d8a3cf1f"}
Nov 01 15:10:30.565888 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:30.565515    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:30.567067 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:30.567039    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7" event={"ID":"d0d5c107-3074-456e-b8a3-c7f7db06291e","Type":"ContainerStarted","Data":"45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca"}
Nov 01 15:10:30.567193 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:30.567087    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7" event={"ID":"d0d5c107-3074-456e-b8a3-c7f7db06291e","Type":"ContainerStarted","Data":"42c3cb3137eafeca40919640b75e86b640f671d425884d12928e8b8942b9e626"}
Nov 01 15:10:30.567193 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:30.567118    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:10:30.683008 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:30.682944    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv" podStartSLOduration=1.682929926 podStartE2EDuration="1.682929926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:30.681566265 +0000 UTC m=+3170.847051564" watchObservedRunningTime="2026-11-01 15:10:30.682929926 +0000 UTC m=+3170.848415247"
Nov 01 15:10:30.813041 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:30.812971    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7" podStartSLOduration=1.8129473759999999 podStartE2EDuration="1.812947376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:30.806940028 +0000 UTC m=+3170.972425327" watchObservedRunningTime="2026-11-01 15:10:30.812947376 +0000 UTC m=+3170.978432676"
Nov 01 15:10:33.511632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:33.511605    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:10:35.149236 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:10:35.149199    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:10:41.576293 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:41.576257    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:10:41.576886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:10:41.576427    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:11:05.149274 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:11:05.149239    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:11:10.332743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.332708    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"]
Nov 01 15:11:10.336761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.336742    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.397639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.397613    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0f1e68-serving-cert\""
Nov 01 15:11:10.426726 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.426702    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.426860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.426831    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8l6g\" (UniqueName: \"kubernetes.io/projected/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-kube-api-access-s8l6g\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.426981 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.426959    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.427028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.427013    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.451306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.451278    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"]
Nov 01 15:11:10.528184 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.528153    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.528311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.528206    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.528311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.528251    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.528311 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:11:10.528295    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-serving-cert: secret "maas-api-e2e-extmod-a-0f1e68-serving-cert" not found
Nov 01 15:11:10.528508 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.528331    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8l6g\" (UniqueName: \"kubernetes.io/projected/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-kube-api-access-s8l6g\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.528508 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:11:10.528350    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls podName:32e9e83a-46d2-4777-9b0f-3418b59d7ee9 nodeName:}" failed. No retries permitted until 2026-11-01 15:11:11.02833504 +0000 UTC m=+3211.193820317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls") pod "maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm" (UID: "32e9e83a-46d2-4777-9b0f-3418b59d7ee9") : secret "maas-api-e2e-extmod-a-0f1e68-serving-cert" not found
Nov 01 15:11:10.528991 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.528965    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.530565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.530545    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:10.587130 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:10.587054    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8l6g\" (UniqueName: \"kubernetes.io/projected/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-kube-api-access-s8l6g\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:11.035495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:11.035362    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:11.038079 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:11.038047    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:11.249608 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:11.249578    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:11.521758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:11.521725    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:11:11.665900 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:11:11.665867    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e9e83a_46d2_4777_9b0f_3418b59d7ee9.slice/crio-9492687dbbf4eca91081b92897f7606fa3884a9a16e5a52689f79bf742ad9680 WatchSource:0}: Error finding container 9492687dbbf4eca91081b92897f7606fa3884a9a16e5a52689f79bf742ad9680: Status 404 returned error can't find the container with id 9492687dbbf4eca91081b92897f7606fa3884a9a16e5a52689f79bf742ad9680
Nov 01 15:11:11.667444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:11.667420    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:11:11.668540 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:11.668518    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"]
Nov 01 15:11:11.816070 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:11.816029    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm" event={"ID":"32e9e83a-46d2-4777-9b0f-3418b59d7ee9","Type":"ContainerStarted","Data":"9492687dbbf4eca91081b92897f7606fa3884a9a16e5a52689f79bf742ad9680"}
Nov 01 15:11:12.823111 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:12.823078    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm" event={"ID":"32e9e83a-46d2-4777-9b0f-3418b59d7ee9","Type":"ContainerStarted","Data":"6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94"}
Nov 01 15:11:12.823566 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:12.823154    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:12.939586 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:12.939494    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm" podStartSLOduration=2.006891712 podStartE2EDuration="2.939480344s" totalImagesPullingTime="932.588632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:11:12.938667013 +0000 UTC m=+3213.104152311" watchObservedRunningTime="2026-11-01 15:11:12.939480344 +0000 UTC m=+3213.104965641"
Nov 01 15:11:18.834500 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:18.834470    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:11:23.843651 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:23.843617    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"]
Nov 01 15:11:23.847844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:23.847825    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:23.900810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:23.900777    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:23.901047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:23.901023    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjmmd\" (UniqueName: \"kubernetes.io/projected/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-kube-api-access-rjmmd\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:23.901135 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:23.901075    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:23.901135 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:23.901108    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:23.902558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:23.902531    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2cf1ee-serving-cert\""
Nov 01 15:11:23.961306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:23.961276    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"]
Nov 01 15:11:24.002773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.002740    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjmmd\" (UniqueName: \"kubernetes.io/projected/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-kube-api-access-rjmmd\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.002936 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.002783    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.002936 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.002878    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.002936 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.002921    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.003128 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:11:24.003050    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-serving-cert: secret "maas-api-e2e-extmod-a-2cf1ee-serving-cert" not found
Nov 01 15:11:24.003128 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:11:24.003115    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls podName:cb41d1da-ceb7-46d0-9abf-b3797c58f3ab nodeName:}" failed. No retries permitted until 2026-11-01 15:11:24.503097482 +0000 UTC m=+3224.668582770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls") pod "maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm" (UID: "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab") : secret "maas-api-e2e-extmod-a-2cf1ee-serving-cert" not found
Nov 01 15:11:24.003646 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.003617    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.005332 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.005313    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.064217 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.064181    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjmmd\" (UniqueName: \"kubernetes.io/projected/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-kube-api-access-rjmmd\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.509375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.509333    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.511937 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.511908    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:24.761156 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:24.761069    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:25.135025 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:11:25.134990    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb41d1da_ceb7_46d0_9abf_b3797c58f3ab.slice/crio-73e1972a7dabbc1f3ef145ba54a77a3263bf52aa25473b31becfcc77abe98855 WatchSource:0}: Error finding container 73e1972a7dabbc1f3ef145ba54a77a3263bf52aa25473b31becfcc77abe98855: Status 404 returned error can't find the container with id 73e1972a7dabbc1f3ef145ba54a77a3263bf52aa25473b31becfcc77abe98855
Nov 01 15:11:25.137946 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:25.137924    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"]
Nov 01 15:11:25.903463 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:25.903417    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm" event={"ID":"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab","Type":"ContainerStarted","Data":"73e1972a7dabbc1f3ef145ba54a77a3263bf52aa25473b31becfcc77abe98855"}
Nov 01 15:11:26.911455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:26.911417    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm" event={"ID":"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab","Type":"ContainerStarted","Data":"f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab"}
Nov 01 15:11:26.911862 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:26.911588    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:27.086058 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:27.086004    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm" podStartSLOduration=3.099345681 podStartE2EDuration="4.085988366s" totalImagesPullingTime="986.642685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:11:27.035872151 +0000 UTC m=+3227.201357448" watchObservedRunningTime="2026-11-01 15:11:27.085988366 +0000 UTC m=+3227.251473660"
Nov 01 15:11:32.923413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:32.923370    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:11:35.153990 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:11:35.153950    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:11:51.220675 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:11:51.220600    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:12:02.154910 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:02.154878    5423 generic.go:356] "Generic (PLEG): container finished" podID="b165a105-0c29-4a2a-800b-76325486d222" containerID="6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907" exitCode=0
Nov 01 15:12:02.155358 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:02.154949    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" event={"ID":"b165a105-0c29-4a2a-800b-76325486d222","Type":"ContainerDied","Data":"6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907"}
Nov 01 15:12:02.155358 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:02.155282    5423 scope.go:122] "RemoveContainer" containerID="6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907"
Nov 01 15:12:02.156764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:02.156739    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:12:02.156840 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:02.156815    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:12:02.156889 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:02.156847    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:12:03.162687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:03.162648    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" event={"ID":"b165a105-0c29-4a2a-800b-76325486d222","Type":"ContainerStarted","Data":"421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914"}
Nov 01 15:12:03.264012 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:03.263978    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:12:03.264952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:03.264929    5423 patch_prober.go:29] interesting pod/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 15:12:03.265069 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:03.264981    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 15:12:04.264841 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:04.264802    5423 patch_prober.go:29] interesting pod/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 15:12:04.265228 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:04.264866    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 15:12:05.150941 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:12:05.150903    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:12:05.264223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:05.264186    5423 patch_prober.go:29] interesting pod/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 15:12:05.264409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:05.264247    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 15:12:06.263925 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:06.263891    5423 patch_prober.go:29] interesting pod/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 15:12:06.264313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:06.263948    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 15:12:07.264833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:07.264797    5423 patch_prober.go:29] interesting pod/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 15:12:07.265239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:07.264866    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 15:12:07.514509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:07.514469    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:12:08.264874 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:08.264837    5423 patch_prober.go:29] interesting pod/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 15:12:08.265419 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:08.264897    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 15:12:09.269376 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:09.269347    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:12:09.270281 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:09.270264    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:12:22.331736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:22.331684    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:12:35.149827 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:12:35.149788    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:12:42.631744 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:42.631713    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:12:42.665520 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:42.665496    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:12:45.324963 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.324924    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"]
Nov 01 15:12:45.328094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.328073    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:45.395983 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.395948    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"]
Nov 01 15:12:45.455440 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.442787    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-15ad1e\""
Nov 01 15:12:45.455440 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.442961    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-15ad1e-dockercfg-ntvc5\""
Nov 01 15:12:45.455440 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.445577    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz\" (UID: \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:45.455440 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.445713    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpzx\" (UniqueName: \"kubernetes.io/projected/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-kube-api-access-kbpzx\") pod \"payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz\" (UID: \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:45.459331 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.458530    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"]
Nov 01 15:12:45.462957 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.462935    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:45.520641 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.520608    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"]
Nov 01 15:12:45.547166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.547134    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz\" (UID: \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:45.547314 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.547175    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2019b212-595e-428a-a511-28d21453ebe9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l\" (UID: \"2019b212-595e-428a-a511-28d21453ebe9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:45.547314 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.547237    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rtmp\" (UniqueName: \"kubernetes.io/projected/2019b212-595e-428a-a511-28d21453ebe9-kube-api-access-6rtmp\") pod \"payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l\" (UID: \"2019b212-595e-428a-a511-28d21453ebe9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:45.547455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.547416    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpzx\" (UniqueName: \"kubernetes.io/projected/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-kube-api-access-kbpzx\") pod \"payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz\" (UID: \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:45.548027 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.548008    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz\" (UID: \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:45.606013 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.605946    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpzx\" (UniqueName: \"kubernetes.io/projected/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-kube-api-access-kbpzx\") pod \"payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz\" (UID: \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:45.640868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.640841    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:45.648297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.648273    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2019b212-595e-428a-a511-28d21453ebe9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l\" (UID: \"2019b212-595e-428a-a511-28d21453ebe9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:45.648433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.648316    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rtmp\" (UniqueName: \"kubernetes.io/projected/2019b212-595e-428a-a511-28d21453ebe9-kube-api-access-6rtmp\") pod \"payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l\" (UID: \"2019b212-595e-428a-a511-28d21453ebe9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:45.648941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.648918    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2019b212-595e-428a-a511-28d21453ebe9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l\" (UID: \"2019b212-595e-428a-a511-28d21453ebe9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:45.710242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.710205    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rtmp\" (UniqueName: \"kubernetes.io/projected/2019b212-595e-428a-a511-28d21453ebe9-kube-api-access-6rtmp\") pod \"payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l\" (UID: \"2019b212-595e-428a-a511-28d21453ebe9\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:45.778450 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:45.778420    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:46.025334 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.025295    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"]
Nov 01 15:12:46.150616 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:12:46.150579    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2019b212_595e_428a_a511_28d21453ebe9.slice/crio-2571bcb8dc3c37a91364d5622997f58e47386331814f9b89b93140908333d501 WatchSource:0}: Error finding container 2571bcb8dc3c37a91364d5622997f58e47386331814f9b89b93140908333d501: Status 404 returned error can't find the container with id 2571bcb8dc3c37a91364d5622997f58e47386331814f9b89b93140908333d501
Nov 01 15:12:46.153638 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.153614    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"]
Nov 01 15:12:46.469463 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.469423    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz" event={"ID":"6555b5a1-eb27-4f67-bc7b-602e739bcc8a","Type":"ContainerStarted","Data":"8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e"}
Nov 01 15:12:46.469958 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.469518    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:46.469958 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.469539    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz" event={"ID":"6555b5a1-eb27-4f67-bc7b-602e739bcc8a","Type":"ContainerStarted","Data":"f28c2b9197f7a52dff7771ed52c3be5cbca79f7b3e095c2fd2a6095969607b51"}
Nov 01 15:12:46.471033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.471003    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l" event={"ID":"2019b212-595e-428a-a511-28d21453ebe9","Type":"ContainerStarted","Data":"2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717"}
Nov 01 15:12:46.471138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.471051    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l" event={"ID":"2019b212-595e-428a-a511-28d21453ebe9","Type":"ContainerStarted","Data":"2571bcb8dc3c37a91364d5622997f58e47386331814f9b89b93140908333d501"}
Nov 01 15:12:46.471138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.471071    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:12:46.587307 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:46.587261    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz" podStartSLOduration=1.587249586 podStartE2EDuration="1.587249586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:12:46.5867198 +0000 UTC m=+3306.752205101" watchObservedRunningTime="2026-11-01 15:12:46.587249586 +0000 UTC m=+3306.752734883"
Nov 01 15:12:49.179886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.179823    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l" podStartSLOduration=4.179807812 podStartE2EDuration="4.179807812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:12:46.713017261 +0000 UTC m=+3306.878502559" watchObservedRunningTime="2026-11-01 15:12:49.179807812 +0000 UTC m=+3309.345293109"
Nov 01 15:12:49.180730 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.180703    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"]
Nov 01 15:12:49.183800 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.183785    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.256448 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.255473    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e92165-openshift-default-dockercfg-slsb6\""
Nov 01 15:12:49.295087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295041    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-certs\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.295265 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295139    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-podinfo\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.295265 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295189    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-token\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.295265 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295233    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmgp9\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-kube-api-access-xmgp9\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.295414 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295283    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-socket\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.295414 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295336    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-data\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.295414 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295364    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istiod-ca-cert\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.295619 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295479    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-envoy\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.295619 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.295530    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-credential-socket\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.350609 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.350570    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"]
Nov 01 15:12:49.396227 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396185    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-podinfo\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396245    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-token\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396285    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmgp9\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-kube-api-access-xmgp9\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396324    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-socket\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396365    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-data\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396565    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istiod-ca-cert\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396719 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396676    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-envoy\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396734    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-credential-socket\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396840 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396822    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-certs\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396903 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396872    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-data\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.396951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.396899    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-socket\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.397134 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.397106    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-credential-socket\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.397304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.397192    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-certs\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.397377 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.397303    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istiod-ca-cert\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.399366 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.399343    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-envoy\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.399635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.399618    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-podinfo\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.455756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.455672    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmgp9\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-kube-api-access-xmgp9\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.455933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.455911    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-token\") pod \"e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") " pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.497521 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.497491    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:49.913254 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:12:49.913219    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95d2c4d_0f27_452d_bf5c_687dd8b95243.slice/crio-27e11d45aab34bfe4c4f2cd4d38035a9c2b4d1175fbe484ac09d649f4b684acd WatchSource:0}: Error finding container 27e11d45aab34bfe4c4f2cd4d38035a9c2b4d1175fbe484ac09d649f4b684acd: Status 404 returned error can't find the container with id 27e11d45aab34bfe4c4f2cd4d38035a9c2b4d1175fbe484ac09d649f4b684acd
Nov 01 15:12:49.916135 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.916105    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:12:49.916220 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.916172    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:12:49.916220 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.916204    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:12:49.916632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:49.916609    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"]
Nov 01 15:12:50.495825 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:50.495784    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" event={"ID":"d95d2c4d-0f27-452d-bf5c-687dd8b95243","Type":"ContainerStarted","Data":"83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd"}
Nov 01 15:12:50.496206 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:50.495851    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" event={"ID":"d95d2c4d-0f27-452d-bf5c-687dd8b95243","Type":"ContainerStarted","Data":"27e11d45aab34bfe4c4f2cd4d38035a9c2b4d1175fbe484ac09d649f4b684acd"}
Nov 01 15:12:50.498213 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:50.498190    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:50.504376 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:50.504355    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:50.620752 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:50.620699    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" podStartSLOduration=1.6206863409999999 podStartE2EDuration="1.620686341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:12:50.618453793 +0000 UTC m=+3310.783939090" watchObservedRunningTime="2026-11-01 15:12:50.620686341 +0000 UTC m=+3310.786171638"
Nov 01 15:12:51.502722 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:51.502692    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:51.503804 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:51.503786    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:12:57.479730 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:57.479698    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:12:57.480257 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:12:57.479766    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:13:02.308229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:02.308190    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:13:05.149785 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:13:05.149737    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:13:09.879949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.879917    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"]
Nov 01 15:13:09.885449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.885426    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.939583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.939553    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c89271-openshift-default-dockercfg-x9s7q\""
Nov 01 15:13:09.993584 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.993556    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"]
Nov 01 15:13:09.996683 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.996649    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-credential-socket\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.996823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.996711    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-data\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.996898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.996833    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-token\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.996964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.996920    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istiod-ca-cert\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.997054 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.997040    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-socket\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.997117 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.997093    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-certs\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.997207 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.997182    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-podinfo\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.997297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.997220    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psvp\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-kube-api-access-5psvp\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:09.997297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:09.997258    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-envoy\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106044    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istiod-ca-cert\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106205    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-socket\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106258    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-certs\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106315    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-podinfo\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106346    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psvp\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-kube-api-access-5psvp\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106412    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-envoy\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106624 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106460    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-credential-socket\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106624 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106523    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-data\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.106624 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.106610    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-token\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.107536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.107512    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istiod-ca-cert\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.110203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.110177    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-socket\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.110651 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.110626    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-certs\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.121409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.117002    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-data\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.136083 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.135963    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-podinfo\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.139454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.136467    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-credential-socket\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.139642 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.139583    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-envoy\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.174857 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.174822    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psvp\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-kube-api-access-5psvp\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.174993 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.174954    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-token\") pod \"e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") " pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.200486 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.200462    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:10.624516 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:13:10.624478    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067e8603_7641_45c7_9d7e_d8cca7b65bd2.slice/crio-2521a1783736c9b05cf417549367f41bc7ea3bf6379bfdf0d192aa407dfe0370 WatchSource:0}: Error finding container 2521a1783736c9b05cf417549367f41bc7ea3bf6379bfdf0d192aa407dfe0370: Status 404 returned error can't find the container with id 2521a1783736c9b05cf417549367f41bc7ea3bf6379bfdf0d192aa407dfe0370
Nov 01 15:13:10.627422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.627342    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:13:10.627536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.627443    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:13:10.627536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.627489    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:13:10.627640 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:10.627595    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"]
Nov 01 15:13:11.625128 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:11.625089    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" event={"ID":"067e8603-7641-45c7-9d7e-d8cca7b65bd2","Type":"ContainerStarted","Data":"7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d"}
Nov 01 15:13:11.625571 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:11.625153    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" event={"ID":"067e8603-7641-45c7-9d7e-d8cca7b65bd2","Type":"ContainerStarted","Data":"2521a1783736c9b05cf417549367f41bc7ea3bf6379bfdf0d192aa407dfe0370"}
Nov 01 15:13:12.201024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:12.200988    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:12.206845 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:12.206817    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:12.327868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:12.327819    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" podStartSLOduration=3.327804869 podStartE2EDuration="3.327804869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:13:11.853767422 +0000 UTC m=+3332.019252720" watchObservedRunningTime="2026-11-01 15:13:12.327804869 +0000 UTC m=+3332.493290166"
Nov 01 15:13:12.631652 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:12.631619    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:12.632657 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:12.632635    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:13:18.506627 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.506581    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"]
Nov 01 15:13:18.517672 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.516694    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:18.562495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.562467    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"]
Nov 01 15:13:18.610374 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.610343    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c89271\""
Nov 01 15:13:18.610532 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.610417    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c89271-dockercfg-qxk62\""
Nov 01 15:13:18.622008 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.621978    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnwh\" (UniqueName: \"kubernetes.io/projected/27723dd8-f628-4727-bccf-15cc3d70a2ef-kube-api-access-7hnwh\") pod \"payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq\" (UID: \"27723dd8-f628-4727-bccf-15cc3d70a2ef\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:18.622142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.622068    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27723dd8-f628-4727-bccf-15cc3d70a2ef-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq\" (UID: \"27723dd8-f628-4727-bccf-15cc3d70a2ef\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:18.723763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.723730    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27723dd8-f628-4727-bccf-15cc3d70a2ef-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq\" (UID: \"27723dd8-f628-4727-bccf-15cc3d70a2ef\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:18.723922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.723907    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnwh\" (UniqueName: \"kubernetes.io/projected/27723dd8-f628-4727-bccf-15cc3d70a2ef-kube-api-access-7hnwh\") pod \"payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq\" (UID: \"27723dd8-f628-4727-bccf-15cc3d70a2ef\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:18.724446 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.724430    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27723dd8-f628-4727-bccf-15cc3d70a2ef-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq\" (UID: \"27723dd8-f628-4727-bccf-15cc3d70a2ef\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:18.782915 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.782843    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnwh\" (UniqueName: \"kubernetes.io/projected/27723dd8-f628-4727-bccf-15cc3d70a2ef-kube-api-access-7hnwh\") pod \"payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq\" (UID: \"27723dd8-f628-4727-bccf-15cc3d70a2ef\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:18.833024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:18.832995    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:19.237143 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:13:19.237112    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27723dd8_f628_4727_bccf_15cc3d70a2ef.slice/crio-ffc4b82c38b2b25e0df69c04c9e4b8695dcecf80c354dce81048f00bee72aa0a WatchSource:0}: Error finding container ffc4b82c38b2b25e0df69c04c9e4b8695dcecf80c354dce81048f00bee72aa0a: Status 404 returned error can't find the container with id ffc4b82c38b2b25e0df69c04c9e4b8695dcecf80c354dce81048f00bee72aa0a
Nov 01 15:13:19.240913 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:19.240869    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"]
Nov 01 15:13:19.676477 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:19.676443    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq" event={"ID":"27723dd8-f628-4727-bccf-15cc3d70a2ef","Type":"ContainerStarted","Data":"deacc78192a4b1746122c59b47842ce375e5e2d378926e037d92b050a37165e7"}
Nov 01 15:13:19.676866 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:19.676509    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq" event={"ID":"27723dd8-f628-4727-bccf-15cc3d70a2ef","Type":"ContainerStarted","Data":"ffc4b82c38b2b25e0df69c04c9e4b8695dcecf80c354dce81048f00bee72aa0a"}
Nov 01 15:13:19.676866 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:19.676616    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:19.794985 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:19.794929    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq" podStartSLOduration=1.794916192 podStartE2EDuration="1.794916192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:13:19.793336999 +0000 UTC m=+3339.958822296" watchObservedRunningTime="2026-11-01 15:13:19.794916192 +0000 UTC m=+3339.960401489"
Nov 01 15:13:30.685518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:13:30.685488    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:13:35.149826 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:13:35.149791    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:14:00.050030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:00.049994    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"]
Nov 01 15:14:00.050612 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:00.050266    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy" containerID="cri-o://4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974" gracePeriod=30
Nov 01 15:14:01.175941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:01.175899    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:14:01.176375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:01.175960    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:14:02.566100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:02.566054    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:14:05.149181 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:14:05.149147    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:14:05.304864 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.304837    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:14:05.399345 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399283    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-istio-token\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-istio-token\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399345 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399319    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/512a93a5-2762-44db-9a84-156bea8770f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512a93a5-2762-44db-9a84-156bea8770f4-istio-podinfo\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399521 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399455    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512a93a5-2762-44db-9a84-156bea8770f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512a93a5-2762-44db-9a84-156bea8770f4-istiod-ca-cert\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399521 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399480    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-kube-api-access-q8xsb\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-kube-api-access-q8xsb\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399610 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399521    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-socket\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399610 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399572    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-credential-socket\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399715 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399616    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-data\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399715 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399661    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-envoy\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399829 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399724    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-certs\") pod \"512a93a5-2762-44db-9a84-156bea8770f4\" (UID: \"512a93a5-2762-44db-9a84-156bea8770f4\") "
Nov 01 15:14:05.399885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399842    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-credential-socket" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:05.399885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399857    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-socket" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:05.399885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.399859    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512a93a5-2762-44db-9a84-156bea8770f4-istiod-ca-cert" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:14:05.400152 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.400117    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-data" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:05.400259 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.400220    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-certs" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:05.400350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.400329    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.400412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.400357    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512a93a5-2762-44db-9a84-156bea8770f4-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.400412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.400374    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.400481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.400415    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.400481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.400432    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.401686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.401651    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-istio-token" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:14:05.401686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.401680    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-kube-api-access-q8xsb" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "kube-api-access-q8xsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:14:05.401996 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.401975    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/512a93a5-2762-44db-9a84-156bea8770f4-istio-podinfo" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:14:05.402054 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.401993    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-envoy" pod "512a93a5-2762-44db-9a84-156bea8770f4" (UID: "512a93a5-2762-44db-9a84-156bea8770f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:05.501632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.501602    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8xsb\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-kube-api-access-q8xsb\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.501769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.501637    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512a93a5-2762-44db-9a84-156bea8770f4-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.501769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.501653    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512a93a5-2762-44db-9a84-156bea8770f4-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.501769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.501665    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512a93a5-2762-44db-9a84-156bea8770f4-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:05.983689 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.983655    5423 generic.go:356] "Generic (PLEG): container finished" podID="512a93a5-2762-44db-9a84-156bea8770f4" containerID="4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974" exitCode=0
Nov 01 15:14:05.983825 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.983702    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"
Nov 01 15:14:05.983825 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.983736    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" event={"ID":"512a93a5-2762-44db-9a84-156bea8770f4","Type":"ContainerDied","Data":"4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974"}
Nov 01 15:14:05.983825 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.983797    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z" event={"ID":"512a93a5-2762-44db-9a84-156bea8770f4","Type":"ContainerDied","Data":"ac34559bec93ffaecd3f2dc28bbc968df62a849ac703d2d9f0e2ebc2dbdc8ae6"}
Nov 01 15:14:05.983988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.983827    5423 scope.go:122] "RemoveContainer" containerID="4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974"
Nov 01 15:14:05.996930 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.996889    5423 scope.go:122] "RemoveContainer" containerID="4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974"
Nov 01 15:14:05.997343 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:14:05.997317    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974\": container with ID starting with 4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974 not found: ID does not exist" containerID="4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974"
Nov 01 15:14:05.997485 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:05.997344    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974"} err="failed to get container status \"4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974\": rpc error: code = NotFound desc = could not find container \"4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974\": container with ID starting with 4266d68048ab4791da1d711e0ec65007aa3242b01d5f0a1ee2307864af357974 not found: ID does not exist"
Nov 01 15:14:06.204024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:06.203990    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"]
Nov 01 15:14:06.255481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:06.255410    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-klx5z"]
Nov 01 15:14:07.961673 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:07.961639    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512a93a5-2762-44db-9a84-156bea8770f4" path="/var/lib/kubelet/pods/512a93a5-2762-44db-9a84-156bea8770f4/volumes"
Nov 01 15:14:30.006094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.006063    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"]
Nov 01 15:14:30.006839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.006817    5423 cpu_manager.go:414] "containerMap: removing container" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy"
Nov 01 15:14:30.006839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.006838    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy"
Nov 01 15:14:30.007007 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.006996    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="512a93a5-2762-44db-9a84-156bea8770f4" containerName="istio-proxy"
Nov 01 15:14:30.010645 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.010627    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:30.093439 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.079785    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-928a13\""
Nov 01 15:14:30.110824 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.110780    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e8e262-4059-44bd-a845-83ffbebadb69-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98\" (UID: \"b5e8e262-4059-44bd-a845-83ffbebadb69\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:30.111021 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.110930    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7db\" (UniqueName: \"kubernetes.io/projected/b5e8e262-4059-44bd-a845-83ffbebadb69-kube-api-access-9g7db\") pod \"payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98\" (UID: \"b5e8e262-4059-44bd-a845-83ffbebadb69\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:30.168951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.160310    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-928a13-dockercfg-69jt4\""
Nov 01 15:14:30.168951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.167081    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"]
Nov 01 15:14:30.196835 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.196801    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"]
Nov 01 15:14:30.204129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.204102    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:30.211879 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.211851    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e8e262-4059-44bd-a845-83ffbebadb69-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98\" (UID: \"b5e8e262-4059-44bd-a845-83ffbebadb69\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:30.211995 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.211980    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7db\" (UniqueName: \"kubernetes.io/projected/b5e8e262-4059-44bd-a845-83ffbebadb69-kube-api-access-9g7db\") pod \"payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98\" (UID: \"b5e8e262-4059-44bd-a845-83ffbebadb69\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:30.212749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.212724    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e8e262-4059-44bd-a845-83ffbebadb69-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98\" (UID: \"b5e8e262-4059-44bd-a845-83ffbebadb69\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:30.242336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.242311    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"]
Nov 01 15:14:30.270045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.269988    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7db\" (UniqueName: \"kubernetes.io/projected/b5e8e262-4059-44bd-a845-83ffbebadb69-kube-api-access-9g7db\") pod \"payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98\" (UID: \"b5e8e262-4059-44bd-a845-83ffbebadb69\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:30.313131 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.313101    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc\" (UID: \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:30.313327 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.313307    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g99z2\" (UniqueName: \"kubernetes.io/projected/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-kube-api-access-g99z2\") pod \"payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc\" (UID: \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:30.324088 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.324058    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:30.414663 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.414628    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc\" (UID: \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:30.414807 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.414793    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99z2\" (UniqueName: \"kubernetes.io/projected/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-kube-api-access-g99z2\") pod \"payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc\" (UID: \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:30.415559 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.415534    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc\" (UID: \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:30.472898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.472875    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99z2\" (UniqueName: \"kubernetes.io/projected/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-kube-api-access-g99z2\") pod \"payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc\" (UID: \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:30.518466 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.518440    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:30.694888 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:14:30.694854    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e8e262_4059_44bd_a845_83ffbebadb69.slice/crio-d95a9e093d024019ea0c5ff108ae302e3ce338f420287e9bbbb56618b0fa821b WatchSource:0}: Error finding container d95a9e093d024019ea0c5ff108ae302e3ce338f420287e9bbbb56618b0fa821b: Status 404 returned error can't find the container with id d95a9e093d024019ea0c5ff108ae302e3ce338f420287e9bbbb56618b0fa821b
Nov 01 15:14:30.697934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.697906    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"]
Nov 01 15:14:30.889258 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:14:30.889227    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee82c73_c5e8_4d23_baba_761ae54fd9e7.slice/crio-4b42f55f1a10181893353a238fb947894310d13717e6ffa9d7b017338305a846 WatchSource:0}: Error finding container 4b42f55f1a10181893353a238fb947894310d13717e6ffa9d7b017338305a846: Status 404 returned error can't find the container with id 4b42f55f1a10181893353a238fb947894310d13717e6ffa9d7b017338305a846
Nov 01 15:14:30.892642 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:30.892618    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"]
Nov 01 15:14:31.189015 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:31.188931    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc" event={"ID":"5ee82c73-c5e8-4d23-baba-761ae54fd9e7","Type":"ContainerStarted","Data":"a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c"}
Nov 01 15:14:31.189015 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:31.188987    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc" event={"ID":"5ee82c73-c5e8-4d23-baba-761ae54fd9e7","Type":"ContainerStarted","Data":"4b42f55f1a10181893353a238fb947894310d13717e6ffa9d7b017338305a846"}
Nov 01 15:14:31.189639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:31.189025    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:31.190614 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:31.190593    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98" event={"ID":"b5e8e262-4059-44bd-a845-83ffbebadb69","Type":"ContainerStarted","Data":"aed7b36814db15874f3ce48a2b810d7c4bf3ea8b2dd3961baa34d27d52f9efb8"}
Nov 01 15:14:31.190738 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:31.190632    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98" event={"ID":"b5e8e262-4059-44bd-a845-83ffbebadb69","Type":"ContainerStarted","Data":"d95a9e093d024019ea0c5ff108ae302e3ce338f420287e9bbbb56618b0fa821b"}
Nov 01 15:14:31.190738 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:31.190666    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:31.306636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:31.306590    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc" podStartSLOduration=1.306577602 podStartE2EDuration="1.306577602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:14:31.30442179 +0000 UTC m=+3411.469907085" watchObservedRunningTime="2026-11-01 15:14:31.306577602 +0000 UTC m=+3411.472062900"
Nov 01 15:14:31.434408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:31.433982    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98" podStartSLOduration=2.43396217 podStartE2EDuration="2.43396217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:14:31.431590651 +0000 UTC m=+3411.597075949" watchObservedRunningTime="2026-11-01 15:14:31.43396217 +0000 UTC m=+3411.599447495"
Nov 01 15:14:35.150009 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:14:35.149965    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:14:42.199757 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:42.199728    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:14:42.200284 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:42.200147    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:14:54.418307 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:14:54.418225    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:15:00.193195 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.193151    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"]
Nov 01 15:15:00.212971 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.212940    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:00.225011 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.224988    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"]
Nov 01 15:15:00.229347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.229325    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"]
Nov 01 15:15:00.229484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.229441    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:00.233935 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.233918    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:00.242452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.242378    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"]
Nov 01 15:15:00.247366 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.247348    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"]
Nov 01 15:15:00.247605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.247462    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:00.253566 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.253545    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:00.262581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.262564    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"]
Nov 01 15:15:00.266480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.266463    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:00.322409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.322259    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a89da3f-9095-493f-bca5-bc75cb4add63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst\" (UID: \"9a89da3f-9095-493f-bca5-bc75cb4add63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:00.322409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.322364    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77vls\" (UniqueName: \"kubernetes.io/projected/9a89da3f-9095-493f-bca5-bc75cb4add63-kube-api-access-77vls\") pod \"maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst\" (UID: \"9a89da3f-9095-493f-bca5-bc75cb4add63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:00.322679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.322442    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/472caed8-b652-471b-a27d-86624fd6cb36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s\" (UID: \"472caed8-b652-471b-a27d-86624fd6cb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:00.322679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.322481    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmznc\" (UniqueName: \"kubernetes.io/projected/3ea08a47-6a32-4942-828f-2c4667b81b60-kube-api-access-cmznc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg\" (UID: \"3ea08a47-6a32-4942-828f-2c4667b81b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:00.322679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.322549    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6lb\" (UniqueName: \"kubernetes.io/projected/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-kube-api-access-gb6lb\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75\" (UID: \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:00.322679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.322616    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qsp\" (UniqueName: \"kubernetes.io/projected/472caed8-b652-471b-a27d-86624fd6cb36-kube-api-access-p6qsp\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s\" (UID: \"472caed8-b652-471b-a27d-86624fd6cb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:00.322679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.322651    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75\" (UID: \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:00.323038 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.322680    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08a47-6a32-4942-828f-2c4667b81b60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg\" (UID: \"3ea08a47-6a32-4942-828f-2c4667b81b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:00.330222 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.330186    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 15:15:00.334991 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.334955    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"]
Nov 01 15:15:00.393064 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.393024    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"]
Nov 01 15:15:00.393259 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.393109    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"]
Nov 01 15:15:00.427226 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427175    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a89da3f-9095-493f-bca5-bc75cb4add63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst\" (UID: \"9a89da3f-9095-493f-bca5-bc75cb4add63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:00.427454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427259    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77vls\" (UniqueName: \"kubernetes.io/projected/9a89da3f-9095-493f-bca5-bc75cb4add63-kube-api-access-77vls\") pod \"maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst\" (UID: \"9a89da3f-9095-493f-bca5-bc75cb4add63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:00.427454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427316    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/472caed8-b652-471b-a27d-86624fd6cb36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s\" (UID: \"472caed8-b652-471b-a27d-86624fd6cb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:00.427454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427354    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmznc\" (UniqueName: \"kubernetes.io/projected/3ea08a47-6a32-4942-828f-2c4667b81b60-kube-api-access-cmznc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg\" (UID: \"3ea08a47-6a32-4942-828f-2c4667b81b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:00.427454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427439    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6lb\" (UniqueName: \"kubernetes.io/projected/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-kube-api-access-gb6lb\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75\" (UID: \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:00.427626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427478    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ce5330-9274-4037-9891-9f3209968341-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw\" (UID: \"b2ce5330-9274-4037-9891-9f3209968341\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:00.427626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427546    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qsp\" (UniqueName: \"kubernetes.io/projected/472caed8-b652-471b-a27d-86624fd6cb36-kube-api-access-p6qsp\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s\" (UID: \"472caed8-b652-471b-a27d-86624fd6cb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:00.427626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427579    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892435-nnrlt\" (UID: \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:00.427626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427615    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75\" (UID: \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:00.427760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427643    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08a47-6a32-4942-828f-2c4667b81b60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg\" (UID: \"3ea08a47-6a32-4942-828f-2c4667b81b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:00.427760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427672    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6fk\" (UniqueName: \"kubernetes.io/projected/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-kube-api-access-nd6fk\") pod \"maas-api-key-cleanup-29892435-nnrlt\" (UID: \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:00.427760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.427751    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vhqw\" (UniqueName: \"kubernetes.io/projected/b2ce5330-9274-4037-9891-9f3209968341-kube-api-access-2vhqw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw\" (UID: \"b2ce5330-9274-4037-9891-9f3209968341\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:00.429016 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.428990    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a89da3f-9095-493f-bca5-bc75cb4add63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst\" (UID: \"9a89da3f-9095-493f-bca5-bc75cb4add63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:00.429462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.429433    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"]
Nov 01 15:15:00.429845 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.429828    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08a47-6a32-4942-828f-2c4667b81b60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg\" (UID: \"3ea08a47-6a32-4942-828f-2c4667b81b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:00.430079 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.430056    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75\" (UID: \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:00.430257 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.430240    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/472caed8-b652-471b-a27d-86624fd6cb36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s\" (UID: \"472caed8-b652-471b-a27d-86624fd6cb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:00.452548 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.452470    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"]
Nov 01 15:15:00.455862 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.455837    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"]
Nov 01 15:15:00.486586 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.486559    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qsp\" (UniqueName: \"kubernetes.io/projected/472caed8-b652-471b-a27d-86624fd6cb36-kube-api-access-p6qsp\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s\" (UID: \"472caed8-b652-471b-a27d-86624fd6cb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:00.488077 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.488047    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmznc\" (UniqueName: \"kubernetes.io/projected/3ea08a47-6a32-4942-828f-2c4667b81b60-kube-api-access-cmznc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg\" (UID: \"3ea08a47-6a32-4942-828f-2c4667b81b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:00.488338 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.488305    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77vls\" (UniqueName: \"kubernetes.io/projected/9a89da3f-9095-493f-bca5-bc75cb4add63-kube-api-access-77vls\") pod \"maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst\" (UID: \"9a89da3f-9095-493f-bca5-bc75cb4add63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:00.488499 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.488482    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6lb\" (UniqueName: \"kubernetes.io/projected/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-kube-api-access-gb6lb\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75\" (UID: \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:00.529580 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.529550    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ce5330-9274-4037-9891-9f3209968341-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw\" (UID: \"b2ce5330-9274-4037-9891-9f3209968341\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:00.529769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.529614    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892435-nnrlt\" (UID: \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:00.529769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.529651    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6fk\" (UniqueName: \"kubernetes.io/projected/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-kube-api-access-nd6fk\") pod \"maas-api-key-cleanup-29892435-nnrlt\" (UID: \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:00.529769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.529716    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vhqw\" (UniqueName: \"kubernetes.io/projected/b2ce5330-9274-4037-9891-9f3209968341-kube-api-access-2vhqw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw\" (UID: \"b2ce5330-9274-4037-9891-9f3209968341\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:00.530322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.530297    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892435-nnrlt\" (UID: \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:00.530436 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.530328    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ce5330-9274-4037-9891-9f3209968341-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw\" (UID: \"b2ce5330-9274-4037-9891-9f3209968341\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:00.542996 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.542975    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:00.553451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.553434    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:00.558340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.558322    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:00.567983 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.567959    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:00.593052 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.593019    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vhqw\" (UniqueName: \"kubernetes.io/projected/b2ce5330-9274-4037-9891-9f3209968341-kube-api-access-2vhqw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw\" (UID: \"b2ce5330-9274-4037-9891-9f3209968341\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:00.630834 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.630775    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6fk\" (UniqueName: \"kubernetes.io/projected/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-kube-api-access-nd6fk\") pod \"maas-api-key-cleanup-29892435-nnrlt\" (UID: \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:00.873908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.873877    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:00.882183 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.882159    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:00.980050 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:00.974407    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"]
Nov 01 15:15:01.029584 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.029506    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"]
Nov 01 15:15:01.033089 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.032693    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"]
Nov 01 15:15:01.129429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.129400    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"]
Nov 01 15:15:01.220464 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:15:01.220186    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472caed8_b652_471b_a27d_86624fd6cb36.slice/crio-552c6e58fe4eb7eaa7dd81a71922fc00ec3a83f2ae8b4173d7e2ff6b39343fad WatchSource:0}: Error finding container 552c6e58fe4eb7eaa7dd81a71922fc00ec3a83f2ae8b4173d7e2ff6b39343fad: Status 404 returned error can't find the container with id 552c6e58fe4eb7eaa7dd81a71922fc00ec3a83f2ae8b4173d7e2ff6b39343fad
Nov 01 15:15:01.271993 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:15:01.271960    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ca3fe4_66e5_49b3_9f4c_f2254f3378cb.slice/crio-25538c5acccddb6a543b734a9edee9783eb41c70df2051d1d033b6ac431c77ae WatchSource:0}: Error finding container 25538c5acccddb6a543b734a9edee9783eb41c70df2051d1d033b6ac431c77ae: Status 404 returned error can't find the container with id 25538c5acccddb6a543b734a9edee9783eb41c70df2051d1d033b6ac431c77ae
Nov 01 15:15:01.275994 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.275973    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"]
Nov 01 15:15:01.321565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.315513    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"]
Nov 01 15:15:01.454536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.454497    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" event={"ID":"9a89da3f-9095-493f-bca5-bc75cb4add63","Type":"ContainerStarted","Data":"b4a23255444052cd0b4a291ec0d8403e9e40976c6a1642b9bd753698166c121c"}
Nov 01 15:15:01.455915 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.455891    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" event={"ID":"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6","Type":"ContainerStarted","Data":"62a86083b2b147e153c3a00d2142502cfa3b01619f00aa54a1a5aca88399f0d8"}
Nov 01 15:15:01.457148 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.457121    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" event={"ID":"b2ce5330-9274-4037-9891-9f3209968341","Type":"ContainerStarted","Data":"877ee31faf77c4ce59cf6457395544a2eb09bb9ffc0b781e655055b698e0f721"}
Nov 01 15:15:01.458496 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.458475    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" event={"ID":"472caed8-b652-471b-a27d-86624fd6cb36","Type":"ContainerStarted","Data":"552c6e58fe4eb7eaa7dd81a71922fc00ec3a83f2ae8b4173d7e2ff6b39343fad"}
Nov 01 15:15:01.459761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.459740    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" event={"ID":"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb","Type":"ContainerStarted","Data":"25538c5acccddb6a543b734a9edee9783eb41c70df2051d1d033b6ac431c77ae"}
Nov 01 15:15:01.460930 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:01.460907    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" event={"ID":"3ea08a47-6a32-4942-828f-2c4667b81b60","Type":"ContainerStarted","Data":"8130d3ae531c0a43816c8a32dfc1c0426239b092d8185b522314c75719ee5654"}
Nov 01 15:15:02.471934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.471833    5423 generic.go:356] "Generic (PLEG): container finished" podID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerID="a4f764c9e98ef0ed33b6d750f711d81b433e22616173466197623cc14718925d" exitCode=60
Nov 01 15:15:02.471934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.471872    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" event={"ID":"9a89da3f-9095-493f-bca5-bc75cb4add63","Type":"ContainerDied","Data":"a4f764c9e98ef0ed33b6d750f711d81b433e22616173466197623cc14718925d"}
Nov 01 15:15:02.472443 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.472202    5423 scope.go:122] "RemoveContainer" containerID="a4f764c9e98ef0ed33b6d750f711d81b433e22616173466197623cc14718925d"
Nov 01 15:15:02.473744 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.473723    5423 generic.go:356] "Generic (PLEG): container finished" podID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerID="828d064833a9b0eb3e2c07d1aefe70cacce062e5b0048f72ed32bc7095ac9a0a" exitCode=60
Nov 01 15:15:02.473867 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.473808    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" event={"ID":"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6","Type":"ContainerDied","Data":"828d064833a9b0eb3e2c07d1aefe70cacce062e5b0048f72ed32bc7095ac9a0a"}
Nov 01 15:15:02.474007 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.473997    5423 scope.go:122] "RemoveContainer" containerID="828d064833a9b0eb3e2c07d1aefe70cacce062e5b0048f72ed32bc7095ac9a0a"
Nov 01 15:15:02.476569 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.476543    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" event={"ID":"b2ce5330-9274-4037-9891-9f3209968341","Type":"ContainerStarted","Data":"27649d8307bcce1583a382165c96b39eb30cf277220028f0bd29a2e252f6f83a"}
Nov 01 15:15:02.478254 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.478235    5423 generic.go:356] "Generic (PLEG): container finished" podID="472caed8-b652-471b-a27d-86624fd6cb36" containerID="6e53d8a80f70d4d9fede4792cbeb3489df479947250823c48a799a7ae2f9a5a5" exitCode=60
Nov 01 15:15:02.478369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.478301    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" event={"ID":"472caed8-b652-471b-a27d-86624fd6cb36","Type":"ContainerDied","Data":"6e53d8a80f70d4d9fede4792cbeb3489df479947250823c48a799a7ae2f9a5a5"}
Nov 01 15:15:02.478524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.478509    5423 scope.go:122] "RemoveContainer" containerID="6e53d8a80f70d4d9fede4792cbeb3489df479947250823c48a799a7ae2f9a5a5"
Nov 01 15:15:02.480146 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.480121    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" event={"ID":"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb","Type":"ContainerStarted","Data":"8e90b3be42e25baa947af26723fc52fb6f045d5dae055dce21e0d5ef904961cd"}
Nov 01 15:15:02.482273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.482251    5423 generic.go:356] "Generic (PLEG): container finished" podID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerID="908ad5be65dfb8cea7279340fcdd036d8dffd9a9a4618a41d6ab79a195a40430" exitCode=60
Nov 01 15:15:02.482399 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.482336    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" event={"ID":"3ea08a47-6a32-4942-828f-2c4667b81b60","Type":"ContainerDied","Data":"908ad5be65dfb8cea7279340fcdd036d8dffd9a9a4618a41d6ab79a195a40430"}
Nov 01 15:15:02.482566 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.482535    5423 scope.go:122] "RemoveContainer" containerID="908ad5be65dfb8cea7279340fcdd036d8dffd9a9a4618a41d6ab79a195a40430"
Nov 01 15:15:02.819492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.819443    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" podStartSLOduration=2.819428473 podStartE2EDuration="2.819428473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:02.81879603 +0000 UTC m=+3442.984281401" watchObservedRunningTime="2026-11-01 15:15:02.819428473 +0000 UTC m=+3442.984913822"
Nov 01 15:15:02.933573 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:02.933522    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" podStartSLOduration=2.933509351 podStartE2EDuration="2.933509351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:02.932809649 +0000 UTC m=+3443.098294950" watchObservedRunningTime="2026-11-01 15:15:02.933509351 +0000 UTC m=+3443.098994710"
Nov 01 15:15:03.493706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.493304    5423 generic.go:356] "Generic (PLEG): container finished" podID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerID="8e90b3be42e25baa947af26723fc52fb6f045d5dae055dce21e0d5ef904961cd" exitCode=60
Nov 01 15:15:03.493706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.493349    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" event={"ID":"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb","Type":"ContainerDied","Data":"8e90b3be42e25baa947af26723fc52fb6f045d5dae055dce21e0d5ef904961cd"}
Nov 01 15:15:03.494235 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.493762    5423 scope.go:122] "RemoveContainer" containerID="8e90b3be42e25baa947af26723fc52fb6f045d5dae055dce21e0d5ef904961cd"
Nov 01 15:15:03.497828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.497234    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" event={"ID":"3ea08a47-6a32-4942-828f-2c4667b81b60","Type":"ContainerStarted","Data":"b76201f790afe60e22c8fd545db2047d1e2f66e51fa536684d56169d403e7e6e"}
Nov 01 15:15:03.502095 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.502070    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" event={"ID":"9a89da3f-9095-493f-bca5-bc75cb4add63","Type":"ContainerStarted","Data":"ef22faddd0ed4a15ec87098ad5cbb96d190b7afdc420d75caf95acf62110152e"}
Nov 01 15:15:03.506515 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.505961    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" event={"ID":"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6","Type":"ContainerStarted","Data":"331aa424a6306a1860f896da5fc13597c6f2ccc6768b5fd75eca84e077205841"}
Nov 01 15:15:03.513396 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.513361    5423 generic.go:356] "Generic (PLEG): container finished" podID="b2ce5330-9274-4037-9891-9f3209968341" containerID="27649d8307bcce1583a382165c96b39eb30cf277220028f0bd29a2e252f6f83a" exitCode=60
Nov 01 15:15:03.513499 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.513421    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" event={"ID":"b2ce5330-9274-4037-9891-9f3209968341","Type":"ContainerDied","Data":"27649d8307bcce1583a382165c96b39eb30cf277220028f0bd29a2e252f6f83a"}
Nov 01 15:15:03.513706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.513685    5423 scope.go:122] "RemoveContainer" containerID="27649d8307bcce1583a382165c96b39eb30cf277220028f0bd29a2e252f6f83a"
Nov 01 15:15:03.515821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.515797    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" event={"ID":"472caed8-b652-471b-a27d-86624fd6cb36","Type":"ContainerStarted","Data":"5722c461227a709055f5f3fcd38b43559ac0f234912bd822e2aca17868231e65"}
Nov 01 15:15:03.724187 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.724137    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" podStartSLOduration=3.724125593 podStartE2EDuration="3.724125593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:03.722328581 +0000 UTC m=+3443.887813879" watchObservedRunningTime="2026-11-01 15:15:03.724125593 +0000 UTC m=+3443.889610890"
Nov 01 15:15:03.841980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.841925    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" podStartSLOduration=3.841913443 podStartE2EDuration="3.841913443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:03.841729642 +0000 UTC m=+3444.007214964" watchObservedRunningTime="2026-11-01 15:15:03.841913443 +0000 UTC m=+3444.007398740"
Nov 01 15:15:03.957723 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:03.957674    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" podStartSLOduration=3.957660168 podStartE2EDuration="3.957660168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:03.957087301 +0000 UTC m=+3444.122572596" watchObservedRunningTime="2026-11-01 15:15:03.957660168 +0000 UTC m=+3444.123145465"
Nov 01 15:15:04.075417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.075354    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" podStartSLOduration=4.075337316 podStartE2EDuration="4.075337316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:04.072205557 +0000 UTC m=+3444.237690856" watchObservedRunningTime="2026-11-01 15:15:04.075337316 +0000 UTC m=+3444.240822614"
Nov 01 15:15:04.527123 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.527096    5423 generic.go:356] "Generic (PLEG): container finished" podID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerID="b76201f790afe60e22c8fd545db2047d1e2f66e51fa536684d56169d403e7e6e" exitCode=60
Nov 01 15:15:04.527529 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.527184    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" event={"ID":"3ea08a47-6a32-4942-828f-2c4667b81b60","Type":"ContainerDied","Data":"b76201f790afe60e22c8fd545db2047d1e2f66e51fa536684d56169d403e7e6e"}
Nov 01 15:15:04.527529 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.527250    5423 scope.go:122] "RemoveContainer" containerID="908ad5be65dfb8cea7279340fcdd036d8dffd9a9a4618a41d6ab79a195a40430"
Nov 01 15:15:04.527529 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.527489    5423 scope.go:122] "RemoveContainer" containerID="b76201f790afe60e22c8fd545db2047d1e2f66e51fa536684d56169d403e7e6e"
Nov 01 15:15:04.527838 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:04.527814    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg_odh-ai-gateway-infra(3ea08a47-6a32-4942-828f-2c4667b81b60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60"
Nov 01 15:15:04.529341 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.529318    5423 generic.go:356] "Generic (PLEG): container finished" podID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerID="ef22faddd0ed4a15ec87098ad5cbb96d190b7afdc420d75caf95acf62110152e" exitCode=60
Nov 01 15:15:04.529491 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.529424    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" event={"ID":"9a89da3f-9095-493f-bca5-bc75cb4add63","Type":"ContainerDied","Data":"ef22faddd0ed4a15ec87098ad5cbb96d190b7afdc420d75caf95acf62110152e"}
Nov 01 15:15:04.529685 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.529671    5423 scope.go:122] "RemoveContainer" containerID="ef22faddd0ed4a15ec87098ad5cbb96d190b7afdc420d75caf95acf62110152e"
Nov 01 15:15:04.529965 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:04.529946    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst_odh-ai-gateway-infra(9a89da3f-9095-493f-bca5-bc75cb4add63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63"
Nov 01 15:15:04.531605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.531579    5423 generic.go:356] "Generic (PLEG): container finished" podID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerID="331aa424a6306a1860f896da5fc13597c6f2ccc6768b5fd75eca84e077205841" exitCode=60
Nov 01 15:15:04.531758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.531668    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" event={"ID":"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6","Type":"ContainerDied","Data":"331aa424a6306a1860f896da5fc13597c6f2ccc6768b5fd75eca84e077205841"}
Nov 01 15:15:04.531910 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.531890    5423 scope.go:122] "RemoveContainer" containerID="331aa424a6306a1860f896da5fc13597c6f2ccc6768b5fd75eca84e077205841"
Nov 01 15:15:04.532275 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:04.532166    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75_odh-ai-gateway-infra(7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6"
Nov 01 15:15:04.533889 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.533864    5423 generic.go:356] "Generic (PLEG): container finished" podID="b2ce5330-9274-4037-9891-9f3209968341" containerID="6931faba6ca546fb44fb8df258d6e9204783874356a2b917b09dd51d6b821d28" exitCode=60
Nov 01 15:15:04.533997 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.533942    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" event={"ID":"b2ce5330-9274-4037-9891-9f3209968341","Type":"ContainerDied","Data":"6931faba6ca546fb44fb8df258d6e9204783874356a2b917b09dd51d6b821d28"}
Nov 01 15:15:04.534209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.534192    5423 scope.go:122] "RemoveContainer" containerID="6931faba6ca546fb44fb8df258d6e9204783874356a2b917b09dd51d6b821d28"
Nov 01 15:15:04.534503 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:04.534477    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw_odh-ai-gateway-infra(b2ce5330-9274-4037-9891-9f3209968341)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" podUID="b2ce5330-9274-4037-9891-9f3209968341"
Nov 01 15:15:04.535930 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.535909    5423 generic.go:356] "Generic (PLEG): container finished" podID="472caed8-b652-471b-a27d-86624fd6cb36" containerID="5722c461227a709055f5f3fcd38b43559ac0f234912bd822e2aca17868231e65" exitCode=60
Nov 01 15:15:04.536033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.535989    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" event={"ID":"472caed8-b652-471b-a27d-86624fd6cb36","Type":"ContainerDied","Data":"5722c461227a709055f5f3fcd38b43559ac0f234912bd822e2aca17868231e65"}
Nov 01 15:15:04.536151 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.536138    5423 scope.go:122] "RemoveContainer" containerID="5722c461227a709055f5f3fcd38b43559ac0f234912bd822e2aca17868231e65"
Nov 01 15:15:04.536400 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:04.536364    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s_odh-ai-gateway-infra(472caed8-b652-471b-a27d-86624fd6cb36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" podUID="472caed8-b652-471b-a27d-86624fd6cb36"
Nov 01 15:15:04.537821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.537799    5423 generic.go:356] "Generic (PLEG): container finished" podID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerID="f5d2e9e18407a43410e1548d38856a0be4995bda7d949ded1f817165667d86e5" exitCode=60
Nov 01 15:15:04.537916 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.537868    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" event={"ID":"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb","Type":"ContainerDied","Data":"f5d2e9e18407a43410e1548d38856a0be4995bda7d949ded1f817165667d86e5"}
Nov 01 15:15:04.538123 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.538107    5423 scope.go:122] "RemoveContainer" containerID="f5d2e9e18407a43410e1548d38856a0be4995bda7d949ded1f817165667d86e5"
Nov 01 15:15:04.538404 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:04.538366    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892435-nnrlt_odh-ai-gateway-infra(80ca3fe4-66e5-49b3-9f4c-f2254f3378cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb"
Nov 01 15:15:04.574559 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.574535    5423 scope.go:122] "RemoveContainer" containerID="a4f764c9e98ef0ed33b6d750f711d81b433e22616173466197623cc14718925d"
Nov 01 15:15:04.588828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.588803    5423 scope.go:122] "RemoveContainer" containerID="828d064833a9b0eb3e2c07d1aefe70cacce062e5b0048f72ed32bc7095ac9a0a"
Nov 01 15:15:04.610519 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.610501    5423 scope.go:122] "RemoveContainer" containerID="27649d8307bcce1583a382165c96b39eb30cf277220028f0bd29a2e252f6f83a"
Nov 01 15:15:04.622804 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.622785    5423 scope.go:122] "RemoveContainer" containerID="6e53d8a80f70d4d9fede4792cbeb3489df479947250823c48a799a7ae2f9a5a5"
Nov 01 15:15:04.635338 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:04.635320    5423 scope.go:122] "RemoveContainer" containerID="8e90b3be42e25baa947af26723fc52fb6f045d5dae055dce21e0d5ef904961cd"
Nov 01 15:15:05.152642 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:05.152594    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:15:05.546154 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.546069    5423 scope.go:122] "RemoveContainer" containerID="b76201f790afe60e22c8fd545db2047d1e2f66e51fa536684d56169d403e7e6e"
Nov 01 15:15:05.546631 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:05.546371    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg_odh-ai-gateway-infra(3ea08a47-6a32-4942-828f-2c4667b81b60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60"
Nov 01 15:15:05.547762 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.547737    5423 scope.go:122] "RemoveContainer" containerID="ef22faddd0ed4a15ec87098ad5cbb96d190b7afdc420d75caf95acf62110152e"
Nov 01 15:15:05.548009 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:05.547982    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst_odh-ai-gateway-infra(9a89da3f-9095-493f-bca5-bc75cb4add63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63"
Nov 01 15:15:05.549273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.549255    5423 scope.go:122] "RemoveContainer" containerID="331aa424a6306a1860f896da5fc13597c6f2ccc6768b5fd75eca84e077205841"
Nov 01 15:15:05.549502 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:05.549484    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75_odh-ai-gateway-infra(7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6"
Nov 01 15:15:05.550497 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.550476    5423 generic.go:356] "Generic (PLEG): container finished" podID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerID="83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd" exitCode=0
Nov 01 15:15:05.550607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.550548    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" event={"ID":"d95d2c4d-0f27-452d-bf5c-687dd8b95243","Type":"ContainerDied","Data":"83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd"}
Nov 01 15:15:05.550878 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.550860    5423 scope.go:122] "RemoveContainer" containerID="83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd"
Nov 01 15:15:05.552434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.552414    5423 scope.go:122] "RemoveContainer" containerID="6931faba6ca546fb44fb8df258d6e9204783874356a2b917b09dd51d6b821d28"
Nov 01 15:15:05.552549 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.552445    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:15:05.552549 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.552524    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:15:05.552673 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.552575    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:15:05.552673 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:05.552619    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw_odh-ai-gateway-infra(b2ce5330-9274-4037-9891-9f3209968341)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" podUID="b2ce5330-9274-4037-9891-9f3209968341"
Nov 01 15:15:05.554063 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.554045    5423 scope.go:122] "RemoveContainer" containerID="5722c461227a709055f5f3fcd38b43559ac0f234912bd822e2aca17868231e65"
Nov 01 15:15:05.554840 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:05.554816    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s_odh-ai-gateway-infra(472caed8-b652-471b-a27d-86624fd6cb36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" podUID="472caed8-b652-471b-a27d-86624fd6cb36"
Nov 01 15:15:05.555527 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:05.555513    5423 scope.go:122] "RemoveContainer" containerID="f5d2e9e18407a43410e1548d38856a0be4995bda7d949ded1f817165667d86e5"
Nov 01 15:15:05.555696 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:05.555678    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892435-nnrlt_odh-ai-gateway-infra(80ca3fe4-66e5-49b3-9f4c-f2254f3378cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb"
Nov 01 15:15:06.509409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:06.505493    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:15:06.564055 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:06.564022    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" event={"ID":"d95d2c4d-0f27-452d-bf5c-687dd8b95243","Type":"ContainerStarted","Data":"7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc"}
Nov 01 15:15:07.519720 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:07.519677    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:15:07.521375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:07.521349    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 15:15:07.521542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:07.521429    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 15:15:08.510806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:08.510769    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 15:15:08.511464 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:08.511430    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 15:15:09.500422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:09.498702    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 15:15:09.500422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:09.498767    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 15:15:10.512410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:10.508765    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 15:15:10.512410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:10.508831    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 15:15:11.503940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:11.503894    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 15:15:11.504135 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:11.503965    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 15:15:12.503542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:12.503510    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:15:12.504013 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:12.503987    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:15:12.504608 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:12.504592    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:15:14.953337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:14.953307    5423 scope.go:122] "RemoveContainer" containerID="ef22faddd0ed4a15ec87098ad5cbb96d190b7afdc420d75caf95acf62110152e"
Nov 01 15:15:14.953853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:14.953564    5423 scope.go:122] "RemoveContainer" containerID="331aa424a6306a1860f896da5fc13597c6f2ccc6768b5fd75eca84e077205841"
Nov 01 15:15:15.634442 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:15.634377    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" event={"ID":"9a89da3f-9095-493f-bca5-bc75cb4add63","Type":"ContainerStarted","Data":"0d8c9caa52010cb068e5e55a2af3b632e03b473d976516dea21497b025afab9f"}
Nov 01 15:15:15.636351 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:15.636324    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" event={"ID":"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6","Type":"ContainerStarted","Data":"70d10085d91ae304dba7b499ad2f7e0b9df14367a36a23e87f533fccf9140d26"}
Nov 01 15:15:15.953274 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:15.953221    5423 scope.go:122] "RemoveContainer" containerID="5722c461227a709055f5f3fcd38b43559ac0f234912bd822e2aca17868231e65"
Nov 01 15:15:16.644926 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.644885    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" event={"ID":"472caed8-b652-471b-a27d-86624fd6cb36","Type":"ContainerStarted","Data":"504965c8df13fb1476b3f7af1731f90b1c1f86bd113dda01c6a37df1b36085b1"}
Nov 01 15:15:16.646879 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.646854    5423 generic.go:356] "Generic (PLEG): container finished" podID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerID="0d8c9caa52010cb068e5e55a2af3b632e03b473d976516dea21497b025afab9f" exitCode=60
Nov 01 15:15:16.647019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.646927    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" event={"ID":"9a89da3f-9095-493f-bca5-bc75cb4add63","Type":"ContainerDied","Data":"0d8c9caa52010cb068e5e55a2af3b632e03b473d976516dea21497b025afab9f"}
Nov 01 15:15:16.647019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.646992    5423 scope.go:122] "RemoveContainer" containerID="ef22faddd0ed4a15ec87098ad5cbb96d190b7afdc420d75caf95acf62110152e"
Nov 01 15:15:16.647309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.647293    5423 scope.go:122] "RemoveContainer" containerID="0d8c9caa52010cb068e5e55a2af3b632e03b473d976516dea21497b025afab9f"
Nov 01 15:15:16.647626 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:16.647602    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst_odh-ai-gateway-infra(9a89da3f-9095-493f-bca5-bc75cb4add63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63"
Nov 01 15:15:16.648775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.648757    5423 generic.go:356] "Generic (PLEG): container finished" podID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerID="70d10085d91ae304dba7b499ad2f7e0b9df14367a36a23e87f533fccf9140d26" exitCode=60
Nov 01 15:15:16.648861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.648794    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" event={"ID":"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6","Type":"ContainerDied","Data":"70d10085d91ae304dba7b499ad2f7e0b9df14367a36a23e87f533fccf9140d26"}
Nov 01 15:15:16.649050 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.649038    5423 scope.go:122] "RemoveContainer" containerID="70d10085d91ae304dba7b499ad2f7e0b9df14367a36a23e87f533fccf9140d26"
Nov 01 15:15:16.649251 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:16.649236    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75_odh-ai-gateway-infra(7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6"
Nov 01 15:15:16.659669 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.659656    5423 scope.go:122] "RemoveContainer" containerID="331aa424a6306a1860f896da5fc13597c6f2ccc6768b5fd75eca84e077205841"
Nov 01 15:15:16.760143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.760121    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"]
Nov 01 15:15:16.878437 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.878409    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"]
Nov 01 15:15:16.953829 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:16.953805    5423 scope.go:122] "RemoveContainer" containerID="f5d2e9e18407a43410e1548d38856a0be4995bda7d949ded1f817165667d86e5"
Nov 01 15:15:17.657620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.657590    5423 generic.go:356] "Generic (PLEG): container finished" podID="472caed8-b652-471b-a27d-86624fd6cb36" containerID="504965c8df13fb1476b3f7af1731f90b1c1f86bd113dda01c6a37df1b36085b1" exitCode=60
Nov 01 15:15:17.658089 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.657664    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" event={"ID":"472caed8-b652-471b-a27d-86624fd6cb36","Type":"ContainerDied","Data":"504965c8df13fb1476b3f7af1731f90b1c1f86bd113dda01c6a37df1b36085b1"}
Nov 01 15:15:17.658089 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.657736    5423 scope.go:122] "RemoveContainer" containerID="5722c461227a709055f5f3fcd38b43559ac0f234912bd822e2aca17868231e65"
Nov 01 15:15:17.658089 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.658073    5423 scope.go:122] "RemoveContainer" containerID="504965c8df13fb1476b3f7af1731f90b1c1f86bd113dda01c6a37df1b36085b1"
Nov 01 15:15:17.658431 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:17.658402    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s_odh-ai-gateway-infra(472caed8-b652-471b-a27d-86624fd6cb36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" podUID="472caed8-b652-471b-a27d-86624fd6cb36"
Nov 01 15:15:17.659683 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.659649    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" event={"ID":"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb","Type":"ContainerStarted","Data":"6d1c5ddfdd77a9058aa46ff9810aca17f6fb274000a91f9c06faea4f035eeda3"}
Nov 01 15:15:17.774553 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.774521    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"]
Nov 01 15:15:17.839553 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.839519    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:17.883946 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.883924    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:17.954773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.954100    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a89da3f-9095-493f-bca5-bc75cb4add63-kube-api-access-77vls\" (UniqueName: \"kubernetes.io/projected/9a89da3f-9095-493f-bca5-bc75cb4add63-kube-api-access-77vls\") pod \"9a89da3f-9095-493f-bca5-bc75cb4add63\" (UID: \"9a89da3f-9095-493f-bca5-bc75cb4add63\") "
Nov 01 15:15:17.954773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.954162    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a89da3f-9095-493f-bca5-bc75cb4add63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a89da3f-9095-493f-bca5-bc75cb4add63-maas-api-service-ca\") pod \"9a89da3f-9095-493f-bca5-bc75cb4add63\" (UID: \"9a89da3f-9095-493f-bca5-bc75cb4add63\") "
Nov 01 15:15:17.954773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.954229    5423 scope.go:122] "RemoveContainer" containerID="b76201f790afe60e22c8fd545db2047d1e2f66e51fa536684d56169d403e7e6e"
Nov 01 15:15:17.954773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.954680    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a89da3f-9095-493f-bca5-bc75cb4add63-maas-api-service-ca" pod "9a89da3f-9095-493f-bca5-bc75cb4add63" (UID: "9a89da3f-9095-493f-bca5-bc75cb4add63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:15:17.955118 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.955037    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-maas-api-service-ca\") pod \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\" (UID: \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\") "
Nov 01 15:15:17.955543 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.955137    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-kube-api-access-gb6lb\" (UniqueName: \"kubernetes.io/projected/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-kube-api-access-gb6lb\") pod \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\" (UID: \"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6\") "
Nov 01 15:15:17.955543 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.955474    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-maas-api-service-ca" pod "7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" (UID: "7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:15:17.955839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.955819    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a89da3f-9095-493f-bca5-bc75cb4add63-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:17.955949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.955843    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:17.956584 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.956463    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a89da3f-9095-493f-bca5-bc75cb4add63-kube-api-access-77vls" pod "9a89da3f-9095-493f-bca5-bc75cb4add63" (UID: "9a89da3f-9095-493f-bca5-bc75cb4add63"). InnerVolumeSpecName "kube-api-access-77vls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:15:17.957513 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:17.957491    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-kube-api-access-gb6lb" pod "7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" (UID: "7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6"). InnerVolumeSpecName "kube-api-access-gb6lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:15:18.057711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.057679    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77vls\" (UniqueName: \"kubernetes.io/projected/9a89da3f-9095-493f-bca5-bc75cb4add63-kube-api-access-77vls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:18.057711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.057710    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb6lb\" (UniqueName: \"kubernetes.io/projected/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6-kube-api-access-gb6lb\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:18.673900 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.673870    5423 generic.go:356] "Generic (PLEG): container finished" podID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerID="6d1c5ddfdd77a9058aa46ff9810aca17f6fb274000a91f9c06faea4f035eeda3" exitCode=60
Nov 01 15:15:18.674318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.673948    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" event={"ID":"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb","Type":"ContainerDied","Data":"6d1c5ddfdd77a9058aa46ff9810aca17f6fb274000a91f9c06faea4f035eeda3"}
Nov 01 15:15:18.674318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.674069    5423 scope.go:122] "RemoveContainer" containerID="f5d2e9e18407a43410e1548d38856a0be4995bda7d949ded1f817165667d86e5"
Nov 01 15:15:18.674504 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.674484    5423 scope.go:122] "RemoveContainer" containerID="6d1c5ddfdd77a9058aa46ff9810aca17f6fb274000a91f9c06faea4f035eeda3"
Nov 01 15:15:18.674839 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:18.674811    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892435-nnrlt_odh-ai-gateway-infra(80ca3fe4-66e5-49b3-9f4c-f2254f3378cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb"
Nov 01 15:15:18.676148 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.676128    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" event={"ID":"3ea08a47-6a32-4942-828f-2c4667b81b60","Type":"ContainerStarted","Data":"887b4f6f224d5e5f678a51ae0e53845f233f983ccc891d70df6c84ac0675c07d"}
Nov 01 15:15:18.677452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.677428    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst" event={"ID":"9a89da3f-9095-493f-bca5-bc75cb4add63","Type":"ContainerDied","Data":"b4a23255444052cd0b4a291ec0d8403e9e40976c6a1642b9bd753698166c121c"}
Nov 01 15:15:18.677452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.677440    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"
Nov 01 15:15:18.678833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.678809    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75" event={"ID":"7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6","Type":"ContainerDied","Data":"62a86083b2b147e153c3a00d2142502cfa3b01619f00aa54a1a5aca88399f0d8"}
Nov 01 15:15:18.678941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.678818    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"
Nov 01 15:15:18.687807 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.687793    5423 scope.go:122] "RemoveContainer" containerID="0d8c9caa52010cb068e5e55a2af3b632e03b473d976516dea21497b025afab9f"
Nov 01 15:15:18.697115 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.697093    5423 scope.go:122] "RemoveContainer" containerID="70d10085d91ae304dba7b499ad2f7e0b9df14367a36a23e87f533fccf9140d26"
Nov 01 15:15:18.847903 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.847872    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:18.918100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.918056    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"]
Nov 01 15:15:18.928315 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.928240    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e92165-29892435-mwzst"]
Nov 01 15:15:18.954122 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.954087    5423 scope.go:122] "RemoveContainer" containerID="6931faba6ca546fb44fb8df258d6e9204783874356a2b917b09dd51d6b821d28"
Nov 01 15:15:18.976961 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.976933    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472caed8-b652-471b-a27d-86624fd6cb36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/472caed8-b652-471b-a27d-86624fd6cb36-maas-api-service-ca\") pod \"472caed8-b652-471b-a27d-86624fd6cb36\" (UID: \"472caed8-b652-471b-a27d-86624fd6cb36\") "
Nov 01 15:15:18.977090 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.976992    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472caed8-b652-471b-a27d-86624fd6cb36-kube-api-access-p6qsp\" (UniqueName: \"kubernetes.io/projected/472caed8-b652-471b-a27d-86624fd6cb36-kube-api-access-p6qsp\") pod \"472caed8-b652-471b-a27d-86624fd6cb36\" (UID: \"472caed8-b652-471b-a27d-86624fd6cb36\") "
Nov 01 15:15:18.979471 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.977362    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472caed8-b652-471b-a27d-86624fd6cb36-maas-api-service-ca" pod "472caed8-b652-471b-a27d-86624fd6cb36" (UID: "472caed8-b652-471b-a27d-86624fd6cb36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:15:18.979471 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.977513    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/472caed8-b652-471b-a27d-86624fd6cb36-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:18.979471 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:18.979459    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472caed8-b652-471b-a27d-86624fd6cb36-kube-api-access-p6qsp" pod "472caed8-b652-471b-a27d-86624fd6cb36" (UID: "472caed8-b652-471b-a27d-86624fd6cb36"). InnerVolumeSpecName "kube-api-access-p6qsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:15:19.030124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.030095    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"]
Nov 01 15:15:19.079339 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.079271    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6qsp\" (UniqueName: \"kubernetes.io/projected/472caed8-b652-471b-a27d-86624fd6cb36-kube-api-access-p6qsp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:19.157012 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.156976    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"]
Nov 01 15:15:19.164444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.164413    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f1e68-29892435-vfp75"]
Nov 01 15:15:19.689034 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.689000    5423 generic.go:356] "Generic (PLEG): container finished" podID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerID="887b4f6f224d5e5f678a51ae0e53845f233f983ccc891d70df6c84ac0675c07d" exitCode=60
Nov 01 15:15:19.689621 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.689077    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" event={"ID":"3ea08a47-6a32-4942-828f-2c4667b81b60","Type":"ContainerDied","Data":"887b4f6f224d5e5f678a51ae0e53845f233f983ccc891d70df6c84ac0675c07d"}
Nov 01 15:15:19.689621 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.689143    5423 scope.go:122] "RemoveContainer" containerID="b76201f790afe60e22c8fd545db2047d1e2f66e51fa536684d56169d403e7e6e"
Nov 01 15:15:19.689621 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.689498    5423 scope.go:122] "RemoveContainer" containerID="887b4f6f224d5e5f678a51ae0e53845f233f983ccc891d70df6c84ac0675c07d"
Nov 01 15:15:19.689851 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:19.689824    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg_odh-ai-gateway-infra(3ea08a47-6a32-4942-828f-2c4667b81b60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60"
Nov 01 15:15:19.693295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.693271    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" event={"ID":"b2ce5330-9274-4037-9891-9f3209968341","Type":"ContainerStarted","Data":"d8955fd5f230d800a4398d0246fda55cd4b9afdc1fa0a374b87b22e5d0291c1a"}
Nov 01 15:15:19.694705 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.694685    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s" event={"ID":"472caed8-b652-471b-a27d-86624fd6cb36","Type":"ContainerDied","Data":"552c6e58fe4eb7eaa7dd81a71922fc00ec3a83f2ae8b4173d7e2ff6b39343fad"}
Nov 01 15:15:19.694819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.694712    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"
Nov 01 15:15:19.703930 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.703907    5423 scope.go:122] "RemoveContainer" containerID="504965c8df13fb1476b3f7af1731f90b1c1f86bd113dda01c6a37df1b36085b1"
Nov 01 15:15:19.923641 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.920537    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"]
Nov 01 15:15:19.924538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.924512    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892435-7vs8s"]
Nov 01 15:15:19.962800 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.962738    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472caed8-b652-471b-a27d-86624fd6cb36" path="/var/lib/kubelet/pods/472caed8-b652-471b-a27d-86624fd6cb36/volumes"
Nov 01 15:15:19.963175 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.963161    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" path="/var/lib/kubelet/pods/7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6/volumes"
Nov 01 15:15:19.963569 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.963557    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" path="/var/lib/kubelet/pods/9a89da3f-9095-493f-bca5-bc75cb4add63/volumes"
Nov 01 15:15:19.983771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:19.983745    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:20.099794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.099360    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-kube-api-access-nd6fk\" (UniqueName: \"kubernetes.io/projected/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-kube-api-access-nd6fk\") pod \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\" (UID: \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\") "
Nov 01 15:15:20.100623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.100218    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-maas-api-service-ca\") pod \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\" (UID: \"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb\") "
Nov 01 15:15:20.101779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.100603    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-maas-api-service-ca" pod "80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" (UID: "80ca3fe4-66e5-49b3-9f4c-f2254f3378cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:15:20.106562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.105770    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-kube-api-access-nd6fk" pod "80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" (UID: "80ca3fe4-66e5-49b3-9f4c-f2254f3378cb"). InnerVolumeSpecName "kube-api-access-nd6fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:15:20.203033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.202993    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd6fk\" (UniqueName: \"kubernetes.io/projected/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-kube-api-access-nd6fk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:20.203033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.203024    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:20.290960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.290873    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"]
Nov 01 15:15:20.702960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.702918    5423 generic.go:356] "Generic (PLEG): container finished" podID="b2ce5330-9274-4037-9891-9f3209968341" containerID="d8955fd5f230d800a4398d0246fda55cd4b9afdc1fa0a374b87b22e5d0291c1a" exitCode=60
Nov 01 15:15:20.703484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.702996    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" event={"ID":"b2ce5330-9274-4037-9891-9f3209968341","Type":"ContainerDied","Data":"d8955fd5f230d800a4398d0246fda55cd4b9afdc1fa0a374b87b22e5d0291c1a"}
Nov 01 15:15:20.703484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.703051    5423 scope.go:122] "RemoveContainer" containerID="6931faba6ca546fb44fb8df258d6e9204783874356a2b917b09dd51d6b821d28"
Nov 01 15:15:20.703915 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.703792    5423 scope.go:122] "RemoveContainer" containerID="d8955fd5f230d800a4398d0246fda55cd4b9afdc1fa0a374b87b22e5d0291c1a"
Nov 01 15:15:20.710491 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:20.704362    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw_odh-ai-gateway-infra(b2ce5330-9274-4037-9891-9f3209968341)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" podUID="b2ce5330-9274-4037-9891-9f3209968341"
Nov 01 15:15:20.715129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.715105    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"
Nov 01 15:15:20.715273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.715124    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt" event={"ID":"80ca3fe4-66e5-49b3-9f4c-f2254f3378cb","Type":"ContainerDied","Data":"25538c5acccddb6a543b734a9edee9783eb41c70df2051d1d033b6ac431c77ae"}
Nov 01 15:15:20.727225 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.727210    5423 scope.go:122] "RemoveContainer" containerID="6d1c5ddfdd77a9058aa46ff9810aca17f6fb274000a91f9c06faea4f035eeda3"
Nov 01 15:15:20.866898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.866868    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:20.932637 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.932601    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"]
Nov 01 15:15:20.935566 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:20.935538    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892435-nnrlt"]
Nov 01 15:15:21.019911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.019805    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea08a47-6a32-4942-828f-2c4667b81b60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08a47-6a32-4942-828f-2c4667b81b60-maas-api-service-ca\") pod \"3ea08a47-6a32-4942-828f-2c4667b81b60\" (UID: \"3ea08a47-6a32-4942-828f-2c4667b81b60\") "
Nov 01 15:15:21.020100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.019945    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea08a47-6a32-4942-828f-2c4667b81b60-kube-api-access-cmznc\" (UniqueName: \"kubernetes.io/projected/3ea08a47-6a32-4942-828f-2c4667b81b60-kube-api-access-cmznc\") pod \"3ea08a47-6a32-4942-828f-2c4667b81b60\" (UID: \"3ea08a47-6a32-4942-828f-2c4667b81b60\") "
Nov 01 15:15:21.020286 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.020195    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea08a47-6a32-4942-828f-2c4667b81b60-maas-api-service-ca" pod "3ea08a47-6a32-4942-828f-2c4667b81b60" (UID: "3ea08a47-6a32-4942-828f-2c4667b81b60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:15:21.020988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.020961    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08a47-6a32-4942-828f-2c4667b81b60-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:21.022480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.022443    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea08a47-6a32-4942-828f-2c4667b81b60-kube-api-access-cmznc" pod "3ea08a47-6a32-4942-828f-2c4667b81b60" (UID: "3ea08a47-6a32-4942-828f-2c4667b81b60"). InnerVolumeSpecName "kube-api-access-cmznc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:15:21.050557 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.050524    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"]
Nov 01 15:15:21.124138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.124087    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmznc\" (UniqueName: \"kubernetes.io/projected/3ea08a47-6a32-4942-828f-2c4667b81b60-kube-api-access-cmznc\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:21.731833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.731805    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"
Nov 01 15:15:21.732256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.731853    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg" event={"ID":"3ea08a47-6a32-4942-828f-2c4667b81b60","Type":"ContainerDied","Data":"8130d3ae531c0a43816c8a32dfc1c0426239b092d8185b522314c75719ee5654"}
Nov 01 15:15:21.732256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.731913    5423 scope.go:122] "RemoveContainer" containerID="887b4f6f224d5e5f678a51ae0e53845f233f983ccc891d70df6c84ac0675c07d"
Nov 01 15:15:21.927455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.924806    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:21.961959 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.961926    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" path="/var/lib/kubelet/pods/80ca3fe4-66e5-49b3-9f4c-f2254f3378cb/volumes"
Nov 01 15:15:21.964557 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.964538    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"]
Nov 01 15:15:21.966922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:21.966902    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892435-2v2xg"]
Nov 01 15:15:22.035304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.035222    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ce5330-9274-4037-9891-9f3209968341-kube-api-access-2vhqw\" (UniqueName: \"kubernetes.io/projected/b2ce5330-9274-4037-9891-9f3209968341-kube-api-access-2vhqw\") pod \"b2ce5330-9274-4037-9891-9f3209968341\" (UID: \"b2ce5330-9274-4037-9891-9f3209968341\") "
Nov 01 15:15:22.035522 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.035331    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ce5330-9274-4037-9891-9f3209968341-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ce5330-9274-4037-9891-9f3209968341-maas-api-service-ca\") pod \"b2ce5330-9274-4037-9891-9f3209968341\" (UID: \"b2ce5330-9274-4037-9891-9f3209968341\") "
Nov 01 15:15:22.035787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.035759    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ce5330-9274-4037-9891-9f3209968341-maas-api-service-ca" pod "b2ce5330-9274-4037-9891-9f3209968341" (UID: "b2ce5330-9274-4037-9891-9f3209968341"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:15:22.035900 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.035886    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ce5330-9274-4037-9891-9f3209968341-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:22.037641 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.037619    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ce5330-9274-4037-9891-9f3209968341-kube-api-access-2vhqw" pod "b2ce5330-9274-4037-9891-9f3209968341" (UID: "b2ce5330-9274-4037-9891-9f3209968341"). InnerVolumeSpecName "kube-api-access-2vhqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:15:22.136697 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.136665    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vhqw\" (UniqueName: \"kubernetes.io/projected/b2ce5330-9274-4037-9891-9f3209968341-kube-api-access-2vhqw\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:22.740922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.740895    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"
Nov 01 15:15:22.741457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.740888    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw" event={"ID":"b2ce5330-9274-4037-9891-9f3209968341","Type":"ContainerDied","Data":"877ee31faf77c4ce59cf6457395544a2eb09bb9ffc0b781e655055b698e0f721"}
Nov 01 15:15:22.741457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.741032    5423 scope.go:122] "RemoveContainer" containerID="d8955fd5f230d800a4398d0246fda55cd4b9afdc1fa0a374b87b22e5d0291c1a"
Nov 01 15:15:22.917313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.917276    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7387-a4dfb3"]
Nov 01 15:15:22.919597 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.919569    5423 cpu_manager.go:414] "containerMap: removing container" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:15:22.919773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.919758    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:15:22.919879 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.919867    5423 cpu_manager.go:414] "containerMap: removing container" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:15:22.919964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.919953    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:15:22.920060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920050    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:15:22.920140 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920129    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:15:22.920223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920214    5423 cpu_manager.go:414] "containerMap: removing container" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:15:22.920302 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920292    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:15:22.920408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920378    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:15:22.920499 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920488    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:15:22.920578 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920568    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.920667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920656    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.920753 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920741    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.920843 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920831    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.920997 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.920985    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:15:22.921193 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921110    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:15:22.921352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921340    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:15:22.921493 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921468    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:15:22.921493 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921496    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921504    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921527    5423 cpu_manager.go:414] "containerMap: removing container" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921536    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921552    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921560    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921574    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921581    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921595    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.921706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921602    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921746    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921759    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921768    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921777    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921788    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921797    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921806    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921815    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921824    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921833    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921844    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921855    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921865    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921875    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921886    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.922188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.921894    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ce5330-9274-4037-9891-9f3209968341" containerName="cleanup"
Nov 01 15:15:22.927255 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.927236    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:22.936160 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.936139    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"]
Nov 01 15:15:22.943414 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.943378    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892435-cvrhw"]
Nov 01 15:15:22.971481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.971454    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-a4dfb3"]
Nov 01 15:15:22.984229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:22.984204    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jngw\""
Nov 01 15:15:23.047756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.047675    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zdr\" (UniqueName: \"kubernetes.io/projected/fd51ee14-d8dd-4aba-85d8-d5a74219ae55-kube-api-access-28zdr\") pod \"test-curl-7387-a4dfb3\" (UID: \"fd51ee14-d8dd-4aba-85d8-d5a74219ae55\") " pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:23.148941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.148906    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zdr\" (UniqueName: \"kubernetes.io/projected/fd51ee14-d8dd-4aba-85d8-d5a74219ae55-kube-api-access-28zdr\") pod \"test-curl-7387-a4dfb3\" (UID: \"fd51ee14-d8dd-4aba-85d8-d5a74219ae55\") " pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:23.223429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.223307    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zdr\" (UniqueName: \"kubernetes.io/projected/fd51ee14-d8dd-4aba-85d8-d5a74219ae55-kube-api-access-28zdr\") pod \"test-curl-7387-a4dfb3\" (UID: \"fd51ee14-d8dd-4aba-85d8-d5a74219ae55\") " pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:23.240292 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.240267    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:23.633297 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:15:23.633264    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd51ee14_d8dd_4aba_85d8_d5a74219ae55.slice/crio-e75ecdaa51c5bfdf84d0ab7beb142b4927e7dd6e593498361d39107c575158d4 WatchSource:0}: Error finding container e75ecdaa51c5bfdf84d0ab7beb142b4927e7dd6e593498361d39107c575158d4: Status 404 returned error can't find the container with id e75ecdaa51c5bfdf84d0ab7beb142b4927e7dd6e593498361d39107c575158d4
Nov 01 15:15:23.636775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.636755    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-a4dfb3"]
Nov 01 15:15:23.749928 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.749893    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-a4dfb3" event={"ID":"fd51ee14-d8dd-4aba-85d8-d5a74219ae55","Type":"ContainerStarted","Data":"1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd"}
Nov 01 15:15:23.750272 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.749940    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-a4dfb3" event={"ID":"fd51ee14-d8dd-4aba-85d8-d5a74219ae55","Type":"ContainerStarted","Data":"e75ecdaa51c5bfdf84d0ab7beb142b4927e7dd6e593498361d39107c575158d4"}
Nov 01 15:15:23.881274 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.881159    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7387-a4dfb3" podStartSLOduration=1.881145021 podStartE2EDuration="1.881145021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:23.879893739 +0000 UTC m=+3464.045379038" watchObservedRunningTime="2026-11-01 15:15:23.881145021 +0000 UTC m=+3464.046630323"
Nov 01 15:15:23.961363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.961319    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea08a47-6a32-4942-828f-2c4667b81b60" path="/var/lib/kubelet/pods/3ea08a47-6a32-4942-828f-2c4667b81b60/volumes"
Nov 01 15:15:23.961866 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:23.961842    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ce5330-9274-4037-9891-9f3209968341" path="/var/lib/kubelet/pods/b2ce5330-9274-4037-9891-9f3209968341/volumes"
Nov 01 15:15:25.458863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:25.458830    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7387-a4dfb3"]
Nov 01 15:15:25.463542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:25.463502    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7387-a4dfb3"]
Nov 01 15:15:25.762592 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:25.762511    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7387-a4dfb3" podUID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" containerName="test-curl-7387-a4dfb3" containerID="cri-o://1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd" gracePeriod=2
Nov 01 15:15:25.820526 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:25.820476    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-a4dfb3\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:28.016562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.016491    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:28.107933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.107892    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd51ee14-d8dd-4aba-85d8-d5a74219ae55-kube-api-access-28zdr\" (UniqueName: \"kubernetes.io/projected/fd51ee14-d8dd-4aba-85d8-d5a74219ae55-kube-api-access-28zdr\") pod \"fd51ee14-d8dd-4aba-85d8-d5a74219ae55\" (UID: \"fd51ee14-d8dd-4aba-85d8-d5a74219ae55\") "
Nov 01 15:15:28.110206 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.110174    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd51ee14-d8dd-4aba-85d8-d5a74219ae55-kube-api-access-28zdr" pod "fd51ee14-d8dd-4aba-85d8-d5a74219ae55" (UID: "fd51ee14-d8dd-4aba-85d8-d5a74219ae55"). InnerVolumeSpecName "kube-api-access-28zdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:15:28.209745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.209712    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zdr\" (UniqueName: \"kubernetes.io/projected/fd51ee14-d8dd-4aba-85d8-d5a74219ae55-kube-api-access-28zdr\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:28.674548 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.674503    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:15:28.674733 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.674586    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:15:28.788318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.788283    5423 generic.go:356] "Generic (PLEG): container finished" podID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" containerID="1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd" exitCode=137
Nov 01 15:15:28.788543 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.788326    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:28.788543 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.788340    5423 scope.go:122] "RemoveContainer" containerID="1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd"
Nov 01 15:15:28.800326 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.800308    5423 scope.go:122] "RemoveContainer" containerID="1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd"
Nov 01 15:15:28.800659 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:28.800640    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd\": container with ID starting with 1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd not found: ID does not exist" containerID="1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd"
Nov 01 15:15:28.800717 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:28.800667    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd"} err="failed to get container status \"1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd\": rpc error: code = NotFound desc = could not find container \"1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd\": container with ID starting with 1ffe6e74d893141e823bd805393f3942b45af9fc4c1d575b55ecc3e69816aabd not found: ID does not exist"
Nov 01 15:15:28.851180 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:28.851141    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-a4dfb3\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:29.963546 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:29.963516    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" path="/var/lib/kubelet/pods/fd51ee14-d8dd-4aba-85d8-d5a74219ae55/volumes"
Nov 01 15:15:30.010324 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:30.010291    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-a4dfb3\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" pod="opendatahub/test-curl-7387-a4dfb3"
Nov 01 15:15:30.804701 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:30.804665    5423 generic.go:356] "Generic (PLEG): container finished" podID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerID="7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d" exitCode=0
Nov 01 15:15:30.804914 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:30.804737    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" event={"ID":"067e8603-7641-45c7-9d7e-d8cca7b65bd2","Type":"ContainerDied","Data":"7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d"}
Nov 01 15:15:30.805203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:30.805189    5423 scope.go:122] "RemoveContainer" containerID="7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d"
Nov 01 15:15:30.806951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:30.806921    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:15:30.807087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:30.807008    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:15:30.807087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:30.807058    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:15:31.826977 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:31.826936    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" event={"ID":"067e8603-7641-45c7-9d7e-d8cca7b65bd2","Type":"ContainerStarted","Data":"780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5"}
Nov 01 15:15:32.208467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:32.205881    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:15:32.208467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:32.206423    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 01 15:15:32.208467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:32.206493    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 01 15:15:33.211412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:33.206739    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 01 15:15:33.211412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:33.206799    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 01 15:15:34.210438 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:34.204419    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 01 15:15:34.210438 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:34.204492    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 01 15:15:35.150054 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:15:35.150019    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:15:35.219776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:35.219738    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 01 15:15:35.219975 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:35.219816    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 01 15:15:36.214410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:36.213807    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 01 15:15:36.214410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:36.213865    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 01 15:15:37.205885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:37.205852    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:15:37.206225 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:37.206202    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:15:37.206731 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:37.206713    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:15:51.102891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:15:51.102861    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:16:01.759058 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759025    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"]
Nov 01 15:16:01.759891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759872    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" containerName="test-curl-7387-a4dfb3"
Nov 01 15:16:01.759956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759892    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" containerName="test-curl-7387-a4dfb3"
Nov 01 15:16:01.759956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759916    5423 cpu_manager.go:414] "containerMap: removing container" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:16:01.759956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759922    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:16:01.759956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759928    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:16:01.759956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759933    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea0b7a5-d566-48c8-89d5-bd1e77d22ad6" containerName="cleanup"
Nov 01 15:16:01.760142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759965    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:16:01.760142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759969    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:16:01.760142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759986    5423 cpu_manager.go:414] "containerMap: removing container" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:16:01.760142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.759990    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ca3fe4-66e5-49b3-9f4c-f2254f3378cb" containerName="cleanup"
Nov 01 15:16:01.760142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.760088    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a89da3f-9095-493f-bca5-bc75cb4add63" containerName="cleanup"
Nov 01 15:16:01.760142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.760103    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd51ee14-d8dd-4aba-85d8-d5a74219ae55" containerName="test-curl-7387-a4dfb3"
Nov 01 15:16:01.760142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.760112    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="472caed8-b652-471b-a27d-86624fd6cb36" containerName="cleanup"
Nov 01 15:16:01.765313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.765291    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:01.821688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.821658    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"]
Nov 01 15:16:01.868482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.868456    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-928a13-serving-cert\""
Nov 01 15:16:01.938019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.937990    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2m7\" (UniqueName: \"kubernetes.io/projected/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-kube-api-access-cq2m7\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:01.938179 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.938061    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-metrics-certs\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:01.938239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.938218    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:01.938296 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:01.938282    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.039288 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.039189    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.039288 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.039281    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.039588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.039324    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2m7\" (UniqueName: \"kubernetes.io/projected/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-kube-api-access-cq2m7\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.039588 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:16:02.039442    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-serving-cert: secret "maas-api-e2e-extmod-b-928a13-serving-cert" not found
Nov 01 15:16:02.039588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.039455    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-metrics-certs\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.039588 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:16:02.039516    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls podName:cae598df-f01c-4c4b-81d2-adb23a5ee6f2 nodeName:}" failed. No retries permitted until 2026-11-01 15:16:02.539494289 +0000 UTC m=+3502.704979573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls") pod "maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95" (UID: "cae598df-f01c-4c4b-81d2-adb23a5ee6f2") : secret "maas-api-e2e-extmod-b-928a13-serving-cert" not found
Nov 01 15:16:02.040339 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.040310    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.042559 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.042528    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-metrics-certs\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.099823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.099796    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2m7\" (UniqueName: \"kubernetes.io/projected/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-kube-api-access-cq2m7\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.545768 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.545733    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.548133 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.548112    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:02.680049 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:02.680024    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:03.051047 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:16:03.051008    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae598df_f01c_4c4b_81d2_adb23a5ee6f2.slice/crio-9457ae9ac4350563b4ccc33d9f65fdc8c8f4c4219bbce0d646b251fb0643250c WatchSource:0}: Error finding container 9457ae9ac4350563b4ccc33d9f65fdc8c8f4c4219bbce0d646b251fb0643250c: Status 404 returned error can't find the container with id 9457ae9ac4350563b4ccc33d9f65fdc8c8f4c4219bbce0d646b251fb0643250c
Nov 01 15:16:03.061613 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:03.059915    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"]
Nov 01 15:16:03.895580 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:03.895544    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"]
Nov 01 15:16:04.050757 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:04.050724    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95" event={"ID":"cae598df-f01c-4c4b-81d2-adb23a5ee6f2","Type":"ContainerStarted","Data":"02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277"}
Nov 01 15:16:04.050954 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:04.050776    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95" event={"ID":"cae598df-f01c-4c4b-81d2-adb23a5ee6f2","Type":"ContainerStarted","Data":"9457ae9ac4350563b4ccc33d9f65fdc8c8f4c4219bbce0d646b251fb0643250c"}
Nov 01 15:16:04.050954 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:04.050798    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:04.169563 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:04.169470    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95" podStartSLOduration=2.320138003 podStartE2EDuration="3.169456581s" totalImagesPullingTime="849.318578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:16:04.166640228 +0000 UTC m=+3504.332125525" watchObservedRunningTime="2026-11-01 15:16:04.169456581 +0000 UTC m=+3504.334941910"
Nov 01 15:16:05.202868 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:16:05.202827    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:16:08.896146 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:08.896099    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn" podUID="78d92a10-1b66-40f0-9744-4e4ea0dc9ead" containerName="payload-pre-processing" containerID="cri-o://62c2d60c15f1e6600209620f8805ff49df53d2b33bca7def6a309d1e4cae940b" gracePeriod=25
Nov 01 15:16:09.087029 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.086997    5423 generic.go:356] "Generic (PLEG): container finished" podID="78d92a10-1b66-40f0-9744-4e4ea0dc9ead" containerID="62c2d60c15f1e6600209620f8805ff49df53d2b33bca7def6a309d1e4cae940b" exitCode=0
Nov 01 15:16:09.087182 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.087084    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn" event={"ID":"78d92a10-1b66-40f0-9744-4e4ea0dc9ead","Type":"ContainerDied","Data":"62c2d60c15f1e6600209620f8805ff49df53d2b33bca7def6a309d1e4cae940b"}
Nov 01 15:16:09.174342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.174315    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:16:09.260847 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.260817    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-kube-api-access-r4vxm\" (UniqueName: \"kubernetes.io/projected/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-kube-api-access-r4vxm\") pod \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\" (UID: \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\") "
Nov 01 15:16:09.261074 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.261055    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-plugins-config-volume\") pod \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\" (UID: \"78d92a10-1b66-40f0-9744-4e4ea0dc9ead\") "
Nov 01 15:16:09.261435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.261412    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-plugins-config-volume" pod "78d92a10-1b66-40f0-9744-4e4ea0dc9ead" (UID: "78d92a10-1b66-40f0-9744-4e4ea0dc9ead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:16:09.261547 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.261531    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:16:09.262896 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.262878    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-kube-api-access-r4vxm" pod "78d92a10-1b66-40f0-9744-4e4ea0dc9ead" (UID: "78d92a10-1b66-40f0-9744-4e4ea0dc9ead"). InnerVolumeSpecName "kube-api-access-r4vxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:16:09.362948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:09.362902    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4vxm\" (UniqueName: \"kubernetes.io/projected/78d92a10-1b66-40f0-9744-4e4ea0dc9ead-kube-api-access-r4vxm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:16:10.063161 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:10.063126    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:16:10.094811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:10.094783    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"
Nov 01 15:16:10.094811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:10.094799    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn" event={"ID":"78d92a10-1b66-40f0-9744-4e4ea0dc9ead","Type":"ContainerDied","Data":"8423411678f51051dc069fa128c62ea51aa02916a69a115688b3ede203e4a404"}
Nov 01 15:16:10.095022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:10.094857    5423 scope.go:122] "RemoveContainer" containerID="62c2d60c15f1e6600209620f8805ff49df53d2b33bca7def6a309d1e4cae940b"
Nov 01 15:16:10.298523 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:10.298435    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"]
Nov 01 15:16:10.301033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:10.301006    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2cf1ee-585f746c4c-szgnn"]
Nov 01 15:16:11.961941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:11.961902    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d92a10-1b66-40f0-9744-4e4ea0dc9ead" path="/var/lib/kubelet/pods/78d92a10-1b66-40f0-9744-4e4ea0dc9ead/volumes"
Nov 01 15:16:25.258378 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:25.258342    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"]
Nov 01 15:16:25.367181 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:25.367150    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"]
Nov 01 15:16:30.259249 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.259204    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv" podUID="039004c4-1357-4304-850b-c0b807ab2da3" containerName="payload-processing" containerID="cri-o://477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995" gracePeriod=25
Nov 01 15:16:30.368307 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.368268    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7" podUID="d0d5c107-3074-456e-b8a3-c7f7db06291e" containerName="payload-pre-processing" containerID="cri-o://45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca" gracePeriod=25
Nov 01 15:16:30.540620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.540597    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:16:30.580318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.580280    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039004c4-1357-4304-850b-c0b807ab2da3-kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/039004c4-1357-4304-850b-c0b807ab2da3-kube-api-access-ccd7w\") pod \"039004c4-1357-4304-850b-c0b807ab2da3\" (UID: \"039004c4-1357-4304-850b-c0b807ab2da3\") "
Nov 01 15:16:30.580519 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.580450    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039004c4-1357-4304-850b-c0b807ab2da3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039004c4-1357-4304-850b-c0b807ab2da3-plugins-config-volume\") pod \"039004c4-1357-4304-850b-c0b807ab2da3\" (UID: \"039004c4-1357-4304-850b-c0b807ab2da3\") "
Nov 01 15:16:30.581045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.581020    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039004c4-1357-4304-850b-c0b807ab2da3-plugins-config-volume" pod "039004c4-1357-4304-850b-c0b807ab2da3" (UID: "039004c4-1357-4304-850b-c0b807ab2da3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:16:30.583492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.583265    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039004c4-1357-4304-850b-c0b807ab2da3-kube-api-access-ccd7w" pod "039004c4-1357-4304-850b-c0b807ab2da3" (UID: "039004c4-1357-4304-850b-c0b807ab2da3"). InnerVolumeSpecName "kube-api-access-ccd7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:16:30.613470 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.613447    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:16:30.682221 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.682182    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d5c107-3074-456e-b8a3-c7f7db06291e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d5c107-3074-456e-b8a3-c7f7db06291e-plugins-config-volume\") pod \"d0d5c107-3074-456e-b8a3-c7f7db06291e\" (UID: \"d0d5c107-3074-456e-b8a3-c7f7db06291e\") "
Nov 01 15:16:30.682444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.682328    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d5c107-3074-456e-b8a3-c7f7db06291e-kube-api-access-ccs5x\" (UniqueName: \"kubernetes.io/projected/d0d5c107-3074-456e-b8a3-c7f7db06291e-kube-api-access-ccs5x\") pod \"d0d5c107-3074-456e-b8a3-c7f7db06291e\" (UID: \"d0d5c107-3074-456e-b8a3-c7f7db06291e\") "
Nov 01 15:16:30.682627 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.682599    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d5c107-3074-456e-b8a3-c7f7db06291e-plugins-config-volume" pod "d0d5c107-3074-456e-b8a3-c7f7db06291e" (UID: "d0d5c107-3074-456e-b8a3-c7f7db06291e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:16:30.682863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.682835    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d5c107-3074-456e-b8a3-c7f7db06291e-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:16:30.682953 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.682863    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039004c4-1357-4304-850b-c0b807ab2da3-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:16:30.682953 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.682880    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/039004c4-1357-4304-850b-c0b807ab2da3-kube-api-access-ccd7w\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:16:30.684700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.684675    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d5c107-3074-456e-b8a3-c7f7db06291e-kube-api-access-ccs5x" pod "d0d5c107-3074-456e-b8a3-c7f7db06291e" (UID: "d0d5c107-3074-456e-b8a3-c7f7db06291e"). InnerVolumeSpecName "kube-api-access-ccs5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:16:30.784018 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:30.783946    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccs5x\" (UniqueName: \"kubernetes.io/projected/d0d5c107-3074-456e-b8a3-c7f7db06291e-kube-api-access-ccs5x\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:16:31.249768 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.249736    5423 generic.go:356] "Generic (PLEG): container finished" podID="039004c4-1357-4304-850b-c0b807ab2da3" containerID="477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995" exitCode=0
Nov 01 15:16:31.249989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.249781    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"
Nov 01 15:16:31.249989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.249821    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv" event={"ID":"039004c4-1357-4304-850b-c0b807ab2da3","Type":"ContainerDied","Data":"477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995"}
Nov 01 15:16:31.249989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.249879    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv" event={"ID":"039004c4-1357-4304-850b-c0b807ab2da3","Type":"ContainerDied","Data":"8fb91f44d72aeca311f6a18a1dd434d3d0ba91dbcf872f33ea9dba90d8a3cf1f"}
Nov 01 15:16:31.249989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.249911    5423 scope.go:122] "RemoveContainer" containerID="477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995"
Nov 01 15:16:31.251557 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.251532    5423 generic.go:356] "Generic (PLEG): container finished" podID="d0d5c107-3074-456e-b8a3-c7f7db06291e" containerID="45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca" exitCode=0
Nov 01 15:16:31.251693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.251590    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"
Nov 01 15:16:31.251693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.251641    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7" event={"ID":"d0d5c107-3074-456e-b8a3-c7f7db06291e","Type":"ContainerDied","Data":"45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca"}
Nov 01 15:16:31.251693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.251676    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7" event={"ID":"d0d5c107-3074-456e-b8a3-c7f7db06291e","Type":"ContainerDied","Data":"42c3cb3137eafeca40919640b75e86b640f671d425884d12928e8b8942b9e626"}
Nov 01 15:16:31.262729 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.262479    5423 scope.go:122] "RemoveContainer" containerID="477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995"
Nov 01 15:16:31.263051 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:16:31.262790    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995\": container with ID starting with 477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995 not found: ID does not exist" containerID="477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995"
Nov 01 15:16:31.263051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.262822    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995"} err="failed to get container status \"477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995\": rpc error: code = NotFound desc = could not find container \"477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995\": container with ID starting with 477d99a8667d3b56dfd693b18cd3e79cf6f0edb080ac9fc51472377836d3a995 not found: ID does not exist"
Nov 01 15:16:31.263051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.262841    5423 scope.go:122] "RemoveContainer" containerID="45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca"
Nov 01 15:16:31.284821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.284795    5423 scope.go:122] "RemoveContainer" containerID="45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca"
Nov 01 15:16:31.285106 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:16:31.285082    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca\": container with ID starting with 45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca not found: ID does not exist" containerID="45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca"
Nov 01 15:16:31.285163 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.285111    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca"} err="failed to get container status \"45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca\": rpc error: code = NotFound desc = could not find container \"45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca\": container with ID starting with 45cd4272597455044ab2b77f70b0952361af0e106d6b5eab636aaa6ac648d6ca not found: ID does not exist"
Nov 01 15:16:31.418085 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.418052    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"]
Nov 01 15:16:31.421542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.421520    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea1620-5cdcc4c98c-w4mb7"]
Nov 01 15:16:31.580941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.580909    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"]
Nov 01 15:16:31.585316 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.585269    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea1620-b5fc66bf-6dvkv"]
Nov 01 15:16:31.961412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.961309    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039004c4-1357-4304-850b-c0b807ab2da3" path="/var/lib/kubelet/pods/039004c4-1357-4304-850b-c0b807ab2da3/volumes"
Nov 01 15:16:31.961908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:31.961892    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d5c107-3074-456e-b8a3-c7f7db06291e" path="/var/lib/kubelet/pods/d0d5c107-3074-456e-b8a3-c7f7db06291e/volumes"
Nov 01 15:16:32.203371 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:16:32.203336    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:16:35.149482 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:16:35.149447    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:17:05.150162 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:05.150121    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:17:05.926524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:05.926493    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"]
Nov 01 15:17:05.926747 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:05.926699    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm" podUID="32e9e83a-46d2-4777-9b0f-3418b59d7ee9" containerName="maas-api" containerID="cri-o://6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94" gracePeriod=30
Nov 01 15:17:06.229883 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.229857    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:17:06.300950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.300922    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-metrics-certs\") pod \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") "
Nov 01 15:17:06.301076 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.300965    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-kube-api-access-s8l6g\" (UniqueName: \"kubernetes.io/projected/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-kube-api-access-s8l6g\") pod \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") "
Nov 01 15:17:06.301076 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.301066    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls\") pod \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") "
Nov 01 15:17:06.301160 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.301109    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-openshift-service-ca\") pod \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\" (UID: \"32e9e83a-46d2-4777-9b0f-3418b59d7ee9\") "
Nov 01 15:17:06.301594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.301557    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-openshift-service-ca" pod "32e9e83a-46d2-4777-9b0f-3418b59d7ee9" (UID: "32e9e83a-46d2-4777-9b0f-3418b59d7ee9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:17:06.303094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.303067    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-metrics-certs" pod "32e9e83a-46d2-4777-9b0f-3418b59d7ee9" (UID: "32e9e83a-46d2-4777-9b0f-3418b59d7ee9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:17:06.303094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.303068    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-kube-api-access-s8l6g" pod "32e9e83a-46d2-4777-9b0f-3418b59d7ee9" (UID: "32e9e83a-46d2-4777-9b0f-3418b59d7ee9"). InnerVolumeSpecName "kube-api-access-s8l6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:17:06.303223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.303148    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls" pod "32e9e83a-46d2-4777-9b0f-3418b59d7ee9" (UID: "32e9e83a-46d2-4777-9b0f-3418b59d7ee9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:17:06.402956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.402917    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:06.402956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.402951    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:06.403166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.402967    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:06.403166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.402985    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8l6g\" (UniqueName: \"kubernetes.io/projected/32e9e83a-46d2-4777-9b0f-3418b59d7ee9-kube-api-access-s8l6g\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:06.504447 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.504340    5423 generic.go:356] "Generic (PLEG): container finished" podID="32e9e83a-46d2-4777-9b0f-3418b59d7ee9" containerID="6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94" exitCode=0
Nov 01 15:17:06.504640 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.504462    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm" event={"ID":"32e9e83a-46d2-4777-9b0f-3418b59d7ee9","Type":"ContainerDied","Data":"6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94"}
Nov 01 15:17:06.504640 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.504546    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm" event={"ID":"32e9e83a-46d2-4777-9b0f-3418b59d7ee9","Type":"ContainerDied","Data":"9492687dbbf4eca91081b92897f7606fa3884a9a16e5a52689f79bf742ad9680"}
Nov 01 15:17:06.504640 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.504582    5423 scope.go:122] "RemoveContainer" containerID="6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94"
Nov 01 15:17:06.504829 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.504466    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"
Nov 01 15:17:06.517718 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.517697    5423 scope.go:122] "RemoveContainer" containerID="6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94"
Nov 01 15:17:06.518189 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:06.518156    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94\": container with ID starting with 6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94 not found: ID does not exist" containerID="6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94"
Nov 01 15:17:06.518299 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.518204    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94"} err="failed to get container status \"6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94\": rpc error: code = NotFound desc = could not find container \"6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94\": container with ID starting with 6bf9b2089a76689689f01350ed2bcf960d0f092faa68fc91d2a739adfc13cb94 not found: ID does not exist"
Nov 01 15:17:06.673737 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.673706    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"]
Nov 01 15:17:06.682916 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:06.682890    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f1e68-57f5b994d6-8zsrm"]
Nov 01 15:17:07.961642 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:07.961609    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e9e83a-46d2-4777-9b0f-3418b59d7ee9" path="/var/lib/kubelet/pods/32e9e83a-46d2-4777-9b0f-3418b59d7ee9/volumes"
Nov 01 15:17:20.606312 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:20.606277    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"]
Nov 01 15:17:20.606830 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:20.606501    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm" podUID="cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" containerName="maas-api" containerID="cri-o://f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab" gracePeriod=30
Nov 01 15:17:20.899279 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:20.899257    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:17:21.001759 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.001719    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-kube-api-access-rjmmd\" (UniqueName: \"kubernetes.io/projected/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-kube-api-access-rjmmd\") pod \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") "
Nov 01 15:17:21.001962 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.001786    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-openshift-service-ca\") pod \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") "
Nov 01 15:17:21.002027 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.002015    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls\") pod \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") "
Nov 01 15:17:21.002081 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.002049    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-metrics-certs\") pod \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\" (UID: \"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab\") "
Nov 01 15:17:21.003986 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.003953    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-openshift-service-ca" pod "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" (UID: "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:17:21.005832 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.005635    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-metrics-certs" pod "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" (UID: "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:17:21.009810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.009782    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls" pod "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" (UID: "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:17:21.011431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.011360    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-kube-api-access-rjmmd" pod "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" (UID: "cb41d1da-ceb7-46d0-9abf-b3797c58f3ab"). InnerVolumeSpecName "kube-api-access-rjmmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:17:21.104363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.104332    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:21.104363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.104361    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:21.104363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.104370    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjmmd\" (UniqueName: \"kubernetes.io/projected/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-kube-api-access-rjmmd\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:21.104603 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.104380    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:21.608508 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.608476    5423 generic.go:356] "Generic (PLEG): container finished" podID="cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" containerID="f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab" exitCode=0
Nov 01 15:17:21.608917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.608525    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"
Nov 01 15:17:21.608917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.608560    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm" event={"ID":"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab","Type":"ContainerDied","Data":"f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab"}
Nov 01 15:17:21.608917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.608618    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm" event={"ID":"cb41d1da-ceb7-46d0-9abf-b3797c58f3ab","Type":"ContainerDied","Data":"73e1972a7dabbc1f3ef145ba54a77a3263bf52aa25473b31becfcc77abe98855"}
Nov 01 15:17:21.608917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.608647    5423 scope.go:122] "RemoveContainer" containerID="f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab"
Nov 01 15:17:21.619716 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.619699    5423 scope.go:122] "RemoveContainer" containerID="f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab"
Nov 01 15:17:21.619964 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:21.619946    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab\": container with ID starting with f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab not found: ID does not exist" containerID="f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab"
Nov 01 15:17:21.620009 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.619973    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab"} err="failed to get container status \"f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab\": rpc error: code = NotFound desc = could not find container \"f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab\": container with ID starting with f6b2e4c5e22ae9eee6366fdd07ac61fef0fed3df76f99a1b17cdc148dd7a52ab not found: ID does not exist"
Nov 01 15:17:21.777216 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.777180    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"]
Nov 01 15:17:21.782501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.782477    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2cf1ee-5f4c546b44-pltrm"]
Nov 01 15:17:21.961887 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:21.961811    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" path="/var/lib/kubelet/pods/cb41d1da-ceb7-46d0-9abf-b3797c58f3ab/volumes"
Nov 01 15:17:23.544498 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:23.544466    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"]
Nov 01 15:17:23.653930 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:23.653902    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"]
Nov 01 15:17:24.702753 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:24.702721    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:17:28.545664 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.545620    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l" podUID="2019b212-595e-428a-a511-28d21453ebe9" containerName="payload-processing" containerID="cri-o://2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717" gracePeriod=25
Nov 01 15:17:28.654556 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.654509    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz" podUID="6555b5a1-eb27-4f67-bc7b-602e739bcc8a" containerName="payload-pre-processing" containerID="cri-o://8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e" gracePeriod=25
Nov 01 15:17:28.819238 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.819208    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:17:28.916168 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.916145    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2019b212-595e-428a-a511-28d21453ebe9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2019b212-595e-428a-a511-28d21453ebe9-plugins-config-volume\") pod \"2019b212-595e-428a-a511-28d21453ebe9\" (UID: \"2019b212-595e-428a-a511-28d21453ebe9\") "
Nov 01 15:17:28.916284 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.916233    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2019b212-595e-428a-a511-28d21453ebe9-kube-api-access-6rtmp\" (UniqueName: \"kubernetes.io/projected/2019b212-595e-428a-a511-28d21453ebe9-kube-api-access-6rtmp\") pod \"2019b212-595e-428a-a511-28d21453ebe9\" (UID: \"2019b212-595e-428a-a511-28d21453ebe9\") "
Nov 01 15:17:28.916609 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.916576    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2019b212-595e-428a-a511-28d21453ebe9-plugins-config-volume" pod "2019b212-595e-428a-a511-28d21453ebe9" (UID: "2019b212-595e-428a-a511-28d21453ebe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:17:28.916767 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.916749    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2019b212-595e-428a-a511-28d21453ebe9-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:28.918183 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.918162    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2019b212-595e-428a-a511-28d21453ebe9-kube-api-access-6rtmp" pod "2019b212-595e-428a-a511-28d21453ebe9" (UID: "2019b212-595e-428a-a511-28d21453ebe9"). InnerVolumeSpecName "kube-api-access-6rtmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:17:28.923574 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:28.923555    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:17:29.017451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.017419    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-kube-api-access-kbpzx\" (UniqueName: \"kubernetes.io/projected/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-kube-api-access-kbpzx\") pod \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\" (UID: \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\") "
Nov 01 15:17:29.017631 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.017618    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-plugins-config-volume\") pod \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\" (UID: \"6555b5a1-eb27-4f67-bc7b-602e739bcc8a\") "
Nov 01 15:17:29.017965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.017943    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-plugins-config-volume" pod "6555b5a1-eb27-4f67-bc7b-602e739bcc8a" (UID: "6555b5a1-eb27-4f67-bc7b-602e739bcc8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:17:29.018063 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.018008    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rtmp\" (UniqueName: \"kubernetes.io/projected/2019b212-595e-428a-a511-28d21453ebe9-kube-api-access-6rtmp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:29.019598 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.019581    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-kube-api-access-kbpzx" pod "6555b5a1-eb27-4f67-bc7b-602e739bcc8a" (UID: "6555b5a1-eb27-4f67-bc7b-602e739bcc8a"). InnerVolumeSpecName "kube-api-access-kbpzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:17:29.119451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.119358    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:29.119451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.119416    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpzx\" (UniqueName: \"kubernetes.io/projected/6555b5a1-eb27-4f67-bc7b-602e739bcc8a-kube-api-access-kbpzx\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:29.659950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.659918    5423 generic.go:356] "Generic (PLEG): container finished" podID="6555b5a1-eb27-4f67-bc7b-602e739bcc8a" containerID="8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e" exitCode=0
Nov 01 15:17:29.660430 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.659968    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"
Nov 01 15:17:29.660430 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.660002    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz" event={"ID":"6555b5a1-eb27-4f67-bc7b-602e739bcc8a","Type":"ContainerDied","Data":"8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e"}
Nov 01 15:17:29.660430 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.660054    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz" event={"ID":"6555b5a1-eb27-4f67-bc7b-602e739bcc8a","Type":"ContainerDied","Data":"f28c2b9197f7a52dff7771ed52c3be5cbca79f7b3e095c2fd2a6095969607b51"}
Nov 01 15:17:29.660430 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.660081    5423 scope.go:122] "RemoveContainer" containerID="8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e"
Nov 01 15:17:29.661434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.661381    5423 generic.go:356] "Generic (PLEG): container finished" podID="2019b212-595e-428a-a511-28d21453ebe9" containerID="2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717" exitCode=0
Nov 01 15:17:29.661489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.661450    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"
Nov 01 15:17:29.661545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.661477    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l" event={"ID":"2019b212-595e-428a-a511-28d21453ebe9","Type":"ContainerDied","Data":"2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717"}
Nov 01 15:17:29.661545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.661526    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l" event={"ID":"2019b212-595e-428a-a511-28d21453ebe9","Type":"ContainerDied","Data":"2571bcb8dc3c37a91364d5622997f58e47386331814f9b89b93140908333d501"}
Nov 01 15:17:29.673371 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.673355    5423 scope.go:122] "RemoveContainer" containerID="8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e"
Nov 01 15:17:29.673657 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:29.673639    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e\": container with ID starting with 8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e not found: ID does not exist" containerID="8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e"
Nov 01 15:17:29.673709 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.673664    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e"} err="failed to get container status \"8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e\": rpc error: code = NotFound desc = could not find container \"8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e\": container with ID starting with 8b5465f404949d8b24a0a0affda347f7db4600fd3d494b4cfc14e3e3d2d22d4e not found: ID does not exist"
Nov 01 15:17:29.673709 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.673679    5423 scope.go:122] "RemoveContainer" containerID="2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717"
Nov 01 15:17:29.682256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.682235    5423 scope.go:122] "RemoveContainer" containerID="2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717"
Nov 01 15:17:29.682541 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:29.682524    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717\": container with ID starting with 2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717 not found: ID does not exist" containerID="2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717"
Nov 01 15:17:29.682606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.682545    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717"} err="failed to get container status \"2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717\": rpc error: code = NotFound desc = could not find container \"2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717\": container with ID starting with 2d1c062064b920334009bf5dda3eb8285ad2951e3ce5cc38c22593cfb4579717 not found: ID does not exist"
Nov 01 15:17:29.828864 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.828791    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"]
Nov 01 15:17:29.879942 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.879912    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-15ad1e-56676f8f55-kxbgz"]
Nov 01 15:17:29.961379 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.961341    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6555b5a1-eb27-4f67-bc7b-602e739bcc8a" path="/var/lib/kubelet/pods/6555b5a1-eb27-4f67-bc7b-602e739bcc8a/volumes"
Nov 01 15:17:29.990831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.990804    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"]
Nov 01 15:17:29.998804 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:29.998780    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-15ad1e-7df4bb885f-8zc9l"]
Nov 01 15:17:31.967725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:31.967691    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2019b212-595e-428a-a511-28d21453ebe9" path="/var/lib/kubelet/pods/2019b212-595e-428a-a511-28d21453ebe9/volumes"
Nov 01 15:17:35.150474 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:35.150440    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:17:42.674272 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:42.674243    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:17:42.709166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:42.709140    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:17:49.408976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:49.408897    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"]
Nov 01 15:17:49.409352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:49.409099    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy" containerID="cri-o://421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914" gracePeriod=30
Nov 01 15:17:51.616823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.616790    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"]
Nov 01 15:17:51.617593 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617571    5423 cpu_manager.go:414] "containerMap: removing container" podUID="32e9e83a-46d2-4777-9b0f-3418b59d7ee9" containerName="maas-api"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617594    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e9e83a-46d2-4777-9b0f-3418b59d7ee9" containerName="maas-api"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617612    5423 cpu_manager.go:414] "containerMap: removing container" podUID="78d92a10-1b66-40f0-9744-4e4ea0dc9ead" containerName="payload-pre-processing"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617620    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d92a10-1b66-40f0-9744-4e4ea0dc9ead" containerName="payload-pre-processing"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617631    5423 cpu_manager.go:414] "containerMap: removing container" podUID="039004c4-1357-4304-850b-c0b807ab2da3" containerName="payload-processing"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617640    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039004c4-1357-4304-850b-c0b807ab2da3" containerName="payload-processing"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617673    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6555b5a1-eb27-4f67-bc7b-602e739bcc8a" containerName="payload-pre-processing"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617681    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6555b5a1-eb27-4f67-bc7b-602e739bcc8a" containerName="payload-pre-processing"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617693    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d0d5c107-3074-456e-b8a3-c7f7db06291e" containerName="payload-pre-processing"
Nov 01 15:17:51.617712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617700    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d5c107-3074-456e-b8a3-c7f7db06291e" containerName="payload-pre-processing"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617729    5423 cpu_manager.go:414] "containerMap: removing container" podUID="2019b212-595e-428a-a511-28d21453ebe9" containerName="payload-processing"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617737    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2019b212-595e-428a-a511-28d21453ebe9" containerName="payload-processing"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617750    5423 cpu_manager.go:414] "containerMap: removing container" podUID="cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" containerName="maas-api"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617760    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" containerName="maas-api"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617906    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e9e83a-46d2-4777-9b0f-3418b59d7ee9" containerName="maas-api"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617920    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d92a10-1b66-40f0-9744-4e4ea0dc9ead" containerName="payload-pre-processing"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617929    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="039004c4-1357-4304-850b-c0b807ab2da3" containerName="payload-processing"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617947    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6555b5a1-eb27-4f67-bc7b-602e739bcc8a" containerName="payload-pre-processing"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617958    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d5c107-3074-456e-b8a3-c7f7db06291e" containerName="payload-pre-processing"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617969    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="2019b212-595e-428a-a511-28d21453ebe9" containerName="payload-processing"
Nov 01 15:17:51.618172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.617979    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb41d1da-ceb7-46d0-9abf-b3797c58f3ab" containerName="maas-api"
Nov 01 15:17:51.621617 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.621595    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.675435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.675405    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-55943a-serving-cert\""
Nov 01 15:17:51.716702 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.716671    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081a7f84-b0fc-46da-afbd-9b58fe9c6322-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.716827 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.716724    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.716827 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.716798    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-metrics-certs\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.716984 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.716972    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5csfr\" (UniqueName: \"kubernetes.io/projected/081a7f84-b0fc-46da-afbd-9b58fe9c6322-kube-api-access-5csfr\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.729051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.729024    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"]
Nov 01 15:17:51.818229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.818198    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5csfr\" (UniqueName: \"kubernetes.io/projected/081a7f84-b0fc-46da-afbd-9b58fe9c6322-kube-api-access-5csfr\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.818380 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.818253    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081a7f84-b0fc-46da-afbd-9b58fe9c6322-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.818380 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.818287    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.818380 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.818305    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-metrics-certs\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.818548 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:51.818443    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-serving-cert: secret "maas-api-e2e-extmod-a-55943a-serving-cert" not found
Nov 01 15:17:51.818548 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:51.818538    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls podName:081a7f84-b0fc-46da-afbd-9b58fe9c6322 nodeName:}" failed. No retries permitted until 2026-11-01 15:17:52.318518681 +0000 UTC m=+3612.484003965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls") pod "maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc" (UID: "081a7f84-b0fc-46da-afbd-9b58fe9c6322") : secret "maas-api-e2e-extmod-a-55943a-serving-cert" not found
Nov 01 15:17:51.819001 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.818970    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081a7f84-b0fc-46da-afbd-9b58fe9c6322-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.820696 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.820675    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-metrics-certs\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:51.886421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:51.886322    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5csfr\" (UniqueName: \"kubernetes.io/projected/081a7f84-b0fc-46da-afbd-9b58fe9c6322-kube-api-access-5csfr\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:52.327220 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:52.327179    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:52.329716 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:52.329695    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls\") pod \"maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:52.542837 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:52.542801    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:52.924321 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:17:52.924285    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081a7f84_b0fc_46da_afbd_9b58fe9c6322.slice/crio-ff66a13e3749ac5bbf9ad9f959dba9afa44f26b7d04ff5da0f1457707c7db6c0 WatchSource:0}: Error finding container ff66a13e3749ac5bbf9ad9f959dba9afa44f26b7d04ff5da0f1457707c7db6c0: Status 404 returned error can't find the container with id ff66a13e3749ac5bbf9ad9f959dba9afa44f26b7d04ff5da0f1457707c7db6c0
Nov 01 15:17:52.925779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:52.925759    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:17:52.927586 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:52.927565    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"]
Nov 01 15:17:53.812556 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:53.812515    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc" event={"ID":"081a7f84-b0fc-46da-afbd-9b58fe9c6322","Type":"ContainerStarted","Data":"ff66a13e3749ac5bbf9ad9f959dba9afa44f26b7d04ff5da0f1457707c7db6c0"}
Nov 01 15:17:54.272524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.272439    5423 patch_prober.go:29] interesting pod/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:17:54.272524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.272495    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:17:54.692425 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.676997    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:17:54.762574 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.762533    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-data\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.762755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.762594    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-istio-token\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-istio-token\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.762755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.762670    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-kube-api-access-pwtnp\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-kube-api-access-pwtnp\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.762755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.762702    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-certs\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.762755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.762729    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-envoy\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.763000 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.762847    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-data" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:17:54.763000 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.762989    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-certs" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:17:54.763155 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763132    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b165a105-0c29-4a2a-800b-76325486d222-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b165a105-0c29-4a2a-800b-76325486d222-istiod-ca-cert\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.763237 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763209    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-socket\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.763237 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763210    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b165a105-0c29-4a2a-800b-76325486d222-istiod-ca-cert" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:17:54.763337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763252    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b165a105-0c29-4a2a-800b-76325486d222-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b165a105-0c29-4a2a-800b-76325486d222-istio-podinfo\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.763337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763304    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-credential-socket\") pod \"b165a105-0c29-4a2a-800b-76325486d222\" (UID: \"b165a105-0c29-4a2a-800b-76325486d222\") "
Nov 01 15:17:54.763494 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763462    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-socket" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:17:54.763680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763658    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-credential-socket" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:17:54.763905 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763879    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.764035 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763910    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.764035 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763925    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.764035 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763939    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.764035 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.763952    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b165a105-0c29-4a2a-800b-76325486d222-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.765211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.765186    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-kube-api-access-pwtnp" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "kube-api-access-pwtnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:17:54.765523 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.765496    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-istio-token" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:17:54.765648 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.765537    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-envoy" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:17:54.765648 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.765575    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b165a105-0c29-4a2a-800b-76325486d222-istio-podinfo" pod "b165a105-0c29-4a2a-800b-76325486d222" (UID: "b165a105-0c29-4a2a-800b-76325486d222"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:17:54.819951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.819926    5423 generic.go:356] "Generic (PLEG): container finished" podID="b165a105-0c29-4a2a-800b-76325486d222" containerID="421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914" exitCode=0
Nov 01 15:17:54.820086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.819974    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"
Nov 01 15:17:54.820086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.820016    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" event={"ID":"b165a105-0c29-4a2a-800b-76325486d222","Type":"ContainerDied","Data":"421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914"}
Nov 01 15:17:54.820086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.820068    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp" event={"ID":"b165a105-0c29-4a2a-800b-76325486d222","Type":"ContainerDied","Data":"715841df8b98a026475f4a77de0467a82b0b89e8b2c280b36e9627435dd9f713"}
Nov 01 15:17:54.820238 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.820099    5423 scope.go:122] "RemoveContainer" containerID="421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914"
Nov 01 15:17:54.821710 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.821686    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc" event={"ID":"081a7f84-b0fc-46da-afbd-9b58fe9c6322","Type":"ContainerStarted","Data":"fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6"}
Nov 01 15:17:54.821844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.821825    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:17:54.830323 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.830304    5423 scope.go:122] "RemoveContainer" containerID="6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907"
Nov 01 15:17:54.839646 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.839630    5423 scope.go:122] "RemoveContainer" containerID="421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914"
Nov 01 15:17:54.839907 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:54.839889    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914\": container with ID starting with 421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914 not found: ID does not exist" containerID="421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914"
Nov 01 15:17:54.839960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.839916    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914"} err="failed to get container status \"421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914\": rpc error: code = NotFound desc = could not find container \"421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914\": container with ID starting with 421c58932d2870e045cda29ab4faad232969ccb2171ba02d89da225aa3889914 not found: ID does not exist"
Nov 01 15:17:54.839960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.839936    5423 scope.go:122] "RemoveContainer" containerID="6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907"
Nov 01 15:17:54.840170 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:17:54.840153    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907\": container with ID starting with 6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907 not found: ID does not exist" containerID="6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907"
Nov 01 15:17:54.840208 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.840177    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907"} err="failed to get container status \"6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907\": rpc error: code = NotFound desc = could not find container \"6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907\": container with ID starting with 6cb16268d07429af796e37105cbb4aa0af25daca99a02fe41cf72ddc249dc907 not found: ID does not exist"
Nov 01 15:17:54.865481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.865446    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.865481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.865476    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtnp\" (UniqueName: \"kubernetes.io/projected/b165a105-0c29-4a2a-800b-76325486d222-kube-api-access-pwtnp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.865653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.865492    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b165a105-0c29-4a2a-800b-76325486d222-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.865653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.865506    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b165a105-0c29-4a2a-800b-76325486d222-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:54.975583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:54.975531    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc" podStartSLOduration=3.00232934 podStartE2EDuration="3.975517788s" totalImagesPullingTime="973.188448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:17:54.973171969 +0000 UTC m=+3615.138657278" watchObservedRunningTime="2026-11-01 15:17:54.975517788 +0000 UTC m=+3615.141003085"
Nov 01 15:17:55.107367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:55.107237    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"]
Nov 01 15:17:55.122166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:55.122129    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea1620-openshift-default-78ccd6cccc-kgjpp"]
Nov 01 15:17:55.746082 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:55.746051    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"]
Nov 01 15:17:55.960582 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:17:55.960552    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b165a105-0c29-4a2a-800b-76325486d222" path="/var/lib/kubelet/pods/b165a105-0c29-4a2a-800b-76325486d222/volumes"
Nov 01 15:18:00.746958 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:00.746915    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq" podUID="27723dd8-f628-4727-bccf-15cc3d70a2ef" containerName="payload-processing" containerID="cri-o://deacc78192a4b1746122c59b47842ce375e5e2d378926e037d92b050a37165e7" gracePeriod=25
Nov 01 15:18:00.845745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:00.845715    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:18:00.869369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:00.869337    5423 generic.go:356] "Generic (PLEG): container finished" podID="27723dd8-f628-4727-bccf-15cc3d70a2ef" containerID="deacc78192a4b1746122c59b47842ce375e5e2d378926e037d92b050a37165e7" exitCode=0
Nov 01 15:18:00.869474 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:00.869434    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq" event={"ID":"27723dd8-f628-4727-bccf-15cc3d70a2ef","Type":"ContainerDied","Data":"deacc78192a4b1746122c59b47842ce375e5e2d378926e037d92b050a37165e7"}
Nov 01 15:18:01.018941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.018882    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:18:01.141739 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.141705    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27723dd8-f628-4727-bccf-15cc3d70a2ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27723dd8-f628-4727-bccf-15cc3d70a2ef-plugins-config-volume\") pod \"27723dd8-f628-4727-bccf-15cc3d70a2ef\" (UID: \"27723dd8-f628-4727-bccf-15cc3d70a2ef\") "
Nov 01 15:18:01.141978 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.141959    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27723dd8-f628-4727-bccf-15cc3d70a2ef-kube-api-access-7hnwh\" (UniqueName: \"kubernetes.io/projected/27723dd8-f628-4727-bccf-15cc3d70a2ef-kube-api-access-7hnwh\") pod \"27723dd8-f628-4727-bccf-15cc3d70a2ef\" (UID: \"27723dd8-f628-4727-bccf-15cc3d70a2ef\") "
Nov 01 15:18:01.142121 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.142099    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27723dd8-f628-4727-bccf-15cc3d70a2ef-plugins-config-volume" pod "27723dd8-f628-4727-bccf-15cc3d70a2ef" (UID: "27723dd8-f628-4727-bccf-15cc3d70a2ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:18:01.142419 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.142374    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27723dd8-f628-4727-bccf-15cc3d70a2ef-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:01.144190 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.144163    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27723dd8-f628-4727-bccf-15cc3d70a2ef-kube-api-access-7hnwh" pod "27723dd8-f628-4727-bccf-15cc3d70a2ef" (UID: "27723dd8-f628-4727-bccf-15cc3d70a2ef"). InnerVolumeSpecName "kube-api-access-7hnwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:18:01.243468 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.243429    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hnwh\" (UniqueName: \"kubernetes.io/projected/27723dd8-f628-4727-bccf-15cc3d70a2ef-kube-api-access-7hnwh\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:01.876253 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.876223    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"
Nov 01 15:18:01.876745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.876267    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq" event={"ID":"27723dd8-f628-4727-bccf-15cc3d70a2ef","Type":"ContainerDied","Data":"ffc4b82c38b2b25e0df69c04c9e4b8695dcecf80c354dce81048f00bee72aa0a"}
Nov 01 15:18:01.876745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:01.876334    5423 scope.go:122] "RemoveContainer" containerID="deacc78192a4b1746122c59b47842ce375e5e2d378926e037d92b050a37165e7"
Nov 01 15:18:02.048548 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:02.048513    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"]
Nov 01 15:18:02.100423 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:02.100371    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c89271-5b5db89889-g9rbq"]
Nov 01 15:18:03.960377 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:03.960348    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27723dd8-f628-4727-bccf-15cc3d70a2ef" path="/var/lib/kubelet/pods/27723dd8-f628-4727-bccf-15cc3d70a2ef/volumes"
Nov 01 15:18:05.149089 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:18:05.149054    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:18:07.707335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:07.707299    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:18:10.374563 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:10.374532    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"]
Nov 01 15:18:10.374967 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:10.374758    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy" containerID="cri-o://780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5" gracePeriod=30
Nov 01 15:18:15.652771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.652708    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:18:15.773458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773422    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-socket\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.773665 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773516    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istiod-ca-cert\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.773665 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773541    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-kube-api-access-5psvp\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-kube-api-access-5psvp\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.773665 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773590    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-certs\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.773665 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773641    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-token\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-token\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.773908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773705    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-envoy\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.773908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773737    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-podinfo\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.773908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773788    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-credential-socket\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.773908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773788    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-socket" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:18:15.773908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773853    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-data\") pod \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\" (UID: \"067e8603-7641-45c7-9d7e-d8cca7b65bd2\") "
Nov 01 15:18:15.774191 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.773986    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-certs" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:18:15.774191 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.774025    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istiod-ca-cert" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:18:15.774322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.774240    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-credential-socket" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:18:15.774322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.774276    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-data" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:18:15.774481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.774460    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.774544 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.774482    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.774544 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.774497    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.774544 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.774510    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.774544 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.774523    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.776167 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.776108    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-envoy" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:18:15.776289 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.776171    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-podinfo" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:18:15.776289 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.776199    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-token" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:18:15.776289 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.776241    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-kube-api-access-5psvp" pod "067e8603-7641-45c7-9d7e-d8cca7b65bd2" (UID: "067e8603-7641-45c7-9d7e-d8cca7b65bd2"). InnerVolumeSpecName "kube-api-access-5psvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:18:15.876030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.875994    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5psvp\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-kube-api-access-5psvp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.876030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.876020    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.876030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.876032    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.876030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.876041    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067e8603-7641-45c7-9d7e-d8cca7b65bd2-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:15.990773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.990738    5423 generic.go:356] "Generic (PLEG): container finished" podID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerID="780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5" exitCode=0
Nov 01 15:18:15.990945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.990866    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"
Nov 01 15:18:15.991033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.990863    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" event={"ID":"067e8603-7641-45c7-9d7e-d8cca7b65bd2","Type":"ContainerDied","Data":"780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5"}
Nov 01 15:18:15.991084 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.991055    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9" event={"ID":"067e8603-7641-45c7-9d7e-d8cca7b65bd2","Type":"ContainerDied","Data":"2521a1783736c9b05cf417549367f41bc7ea3bf6379bfdf0d192aa407dfe0370"}
Nov 01 15:18:15.991121 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:15.991091    5423 scope.go:122] "RemoveContainer" containerID="780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5"
Nov 01 15:18:16.001304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.001280    5423 scope.go:122] "RemoveContainer" containerID="7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d"
Nov 01 15:18:16.013372 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.013350    5423 scope.go:122] "RemoveContainer" containerID="780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5"
Nov 01 15:18:16.013698 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:18:16.013677    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5\": container with ID starting with 780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5 not found: ID does not exist" containerID="780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5"
Nov 01 15:18:16.013788 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.013711    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5"} err="failed to get container status \"780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5\": rpc error: code = NotFound desc = could not find container \"780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5\": container with ID starting with 780b24f9457d54652c33ea87b3483d0c715e23470079da5cdc1e4a87bab0ded5 not found: ID does not exist"
Nov 01 15:18:16.013788 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.013734    5423 scope.go:122] "RemoveContainer" containerID="7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d"
Nov 01 15:18:16.013982 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:18:16.013964    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d\": container with ID starting with 7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d not found: ID does not exist" containerID="7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d"
Nov 01 15:18:16.014046 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.013991    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d"} err="failed to get container status \"7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d\": rpc error: code = NotFound desc = could not find container \"7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d\": container with ID starting with 7da6cc480359fd9061dae922e612f824d75ba13ff612d9ace82a97e30e39833d not found: ID does not exist"
Nov 01 15:18:16.164496 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.164464    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"]
Nov 01 15:18:16.168693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.168664    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c89271-openshift-default-5d865dd6d5-f89t9"]
Nov 01 15:18:16.820395 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.820354    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"]
Nov 01 15:18:16.821061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821040    5423 cpu_manager.go:414] "containerMap: removing container" podUID="27723dd8-f628-4727-bccf-15cc3d70a2ef" containerName="payload-processing"
Nov 01 15:18:16.821061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821059    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27723dd8-f628-4727-bccf-15cc3d70a2ef" containerName="payload-processing"
Nov 01 15:18:16.821159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821086    5423 cpu_manager.go:414] "containerMap: removing container" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy"
Nov 01 15:18:16.821159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821093    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy"
Nov 01 15:18:16.821159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821118    5423 cpu_manager.go:414] "containerMap: removing container" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy"
Nov 01 15:18:16.821159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821124    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy"
Nov 01 15:18:16.821159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821134    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy"
Nov 01 15:18:16.821159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821139    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy"
Nov 01 15:18:16.821352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821225    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy"
Nov 01 15:18:16.821352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821233    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy"
Nov 01 15:18:16.821352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821239    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy"
Nov 01 15:18:16.821352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821248    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="27723dd8-f628-4727-bccf-15cc3d70a2ef" containerName="payload-processing"
Nov 01 15:18:16.821352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.821257    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" containerName="istio-proxy"
Nov 01 15:18:16.824686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.824669    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:16.882105 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.882076    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"]
Nov 01 15:18:16.897357 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.897331    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-provider-credentials\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:16.897503 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.897437    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-routing-overlay-volume\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:16.897558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.897514    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf9f7\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-kube-api-access-mf9f7\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:16.897558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.897550    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-plugins-config-volume\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:16.925832 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.925809    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 15:18:16.930145 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.930126    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 15:18:16.930350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.930337    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 15:18:16.930432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.930400    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-eb52a593c198\""
Nov 01 15:18:16.930496 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.930482    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wm2wx\""
Nov 01 15:18:16.930579 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:16.930567    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 15:18:17.005755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.005719    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-routing-overlay-volume\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.007043 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.007016    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf9f7\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-kube-api-access-mf9f7\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.007264 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.007248    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-plugins-config-volume\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.011095 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.007511    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-provider-credentials\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.013180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.006903    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-routing-overlay-volume\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.013180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.012492    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-plugins-config-volume\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.016531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.016239    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-provider-credentials\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.073575 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.073521    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf9f7\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-kube-api-access-mf9f7\") pod \"payload-processing-external-model-6cb7f88854-2s59w\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") " pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.138101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.138076    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:17.561114 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:18:17.561078    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d37e2d1_e62a_402f_a538_2a05845a2076.slice/crio-5a596a5938227de87c6bd481245bafe45a27d81c211af3b0f9bcd90a0c89049d WatchSource:0}: Error finding container 5a596a5938227de87c6bd481245bafe45a27d81c211af3b0f9bcd90a0c89049d: Status 404 returned error can't find the container with id 5a596a5938227de87c6bd481245bafe45a27d81c211af3b0f9bcd90a0c89049d
Nov 01 15:18:17.563324 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.563298    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"]
Nov 01 15:18:17.961769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:17.961695    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="067e8603-7641-45c7-9d7e-d8cca7b65bd2" path="/var/lib/kubelet/pods/067e8603-7641-45c7-9d7e-d8cca7b65bd2/volumes"
Nov 01 15:18:18.025463 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:18.025423    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w" event={"ID":"6d37e2d1-e62a-402f-a538-2a05845a2076","Type":"ContainerStarted","Data":"a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa"}
Nov 01 15:18:18.025647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:18.025495    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:18.025647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:18.025524    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w" event={"ID":"6d37e2d1-e62a-402f-a538-2a05845a2076","Type":"ContainerStarted","Data":"5a596a5938227de87c6bd481245bafe45a27d81c211af3b0f9bcd90a0c89049d"}
Nov 01 15:18:18.142720 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:18.142670    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w" podStartSLOduration=2.142657236 podStartE2EDuration="2.142657236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:18:18.141523808 +0000 UTC m=+3638.307009106" watchObservedRunningTime="2026-11-01 15:18:18.142657236 +0000 UTC m=+3638.308142527"
Nov 01 15:18:29.035255 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:29.035222    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:18:35.149630 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:18:35.149594    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:18:40.064479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:40.064446    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:18:50.403323 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.403289    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"]
Nov 01 15:18:50.404040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.404017    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy"
Nov 01 15:18:50.404040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.404037    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b165a105-0c29-4a2a-800b-76325486d222" containerName="istio-proxy"
Nov 01 15:18:50.407660 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.407643    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.437565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.437541    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.437678 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.437651    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.437766 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.437750    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.437810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.437778    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.437810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.437806    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.437891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.437830    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.437946 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.437928    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.437984 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.437969    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2gwd\" (UniqueName: \"kubernetes.io/projected/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-kube-api-access-w2gwd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.438080 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.438060    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.460958 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.460938    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58vzm\""
Nov 01 15:18:50.538871 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.538844    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.538871 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.538875    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539076 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539005    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539076 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539041    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539216 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539150    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539275 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539212    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gwd\" (UniqueName: \"kubernetes.io/projected/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-kube-api-access-w2gwd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539289    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539426 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539356    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539504 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539445    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539504 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539497    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539629 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539610    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539718 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539697    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.539910 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.539886    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.540099 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.540044    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.541291 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.541267    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.541775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.541758    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.596854 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.596831    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.597273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.597252    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gwd\" (UniqueName: \"kubernetes.io/projected/d0dd2f89-8808-4b54-8255-e0c4dbd31fe9-kube-api-access-w2gwd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wfr6h\" (UID: \"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:50.613243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.613215    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"]
Nov 01 15:18:50.721087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:50.721011    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:51.140947 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:51.140914    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"]
Nov 01 15:18:51.173900 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:18:51.173869    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dd2f89_8808_4b54_8255_e0c4dbd31fe9.slice/crio-1faf93ed30ca2322027149be9aca7c00ab8751af37863513dce3bf3cab034647 WatchSource:0}: Error finding container 1faf93ed30ca2322027149be9aca7c00ab8751af37863513dce3bf3cab034647: Status 404 returned error can't find the container with id 1faf93ed30ca2322027149be9aca7c00ab8751af37863513dce3bf3cab034647
Nov 01 15:18:51.176535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:51.176499    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:18:51.176647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:51.176575    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:18:51.176647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:51.176606    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:18:51.235399 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:51.235365    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" event={"ID":"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9","Type":"ContainerStarted","Data":"1faf93ed30ca2322027149be9aca7c00ab8751af37863513dce3bf3cab034647"}
Nov 01 15:18:52.244100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:52.244060    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" event={"ID":"d0dd2f89-8808-4b54-8255-e0c4dbd31fe9","Type":"ContainerStarted","Data":"005cb1456b8e582d9ddca71e93582ef5eb47a2bfd1929c2e9dfd49083f2d8f4b"}
Nov 01 15:18:52.461969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:52.461913    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podStartSLOduration=2.461898783 podStartE2EDuration="2.461898783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:18:52.436235627 +0000 UTC m=+3672.601720948" watchObservedRunningTime="2026-11-01 15:18:52.461898783 +0000 UTC m=+3672.627384081"
Nov 01 15:18:52.724412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:52.723596    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:52.725186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:52.725150    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 01 15:18:52.725237 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:52.725208    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 01 15:18:53.721809 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:53.721764    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 01 15:18:53.722355 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:53.721836    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 01 15:18:54.721304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:54.721262    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 01 15:18:54.721531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:54.721317    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 01 15:18:55.722122 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:55.722084    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 01 15:18:55.722557 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:55.722136    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 01 15:18:56.722156 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:56.722022    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 01 15:18:56.722156 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:56.722087    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 01 15:18:57.233030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:57.232985    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"]
Nov 01 15:18:57.233255 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:57.233216    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy" containerID="cri-o://7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc" gracePeriod=30
Nov 01 15:18:57.506877 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:57.506794    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:18:57.506877 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:57.506846    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:18:57.721948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:57.721909    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 01 15:18:57.722147 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:57.721979    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 01 15:18:58.722206 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:58.722167    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 01 15:18:58.722720 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:58.722236    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 01 15:18:59.866061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:59.866023    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:59.866670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:59.866645    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:18:59.870082 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:18:59.870025    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h"
Nov 01 15:19:02.487338 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.487310    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:19:02.635511 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635480    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-data\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.635686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635516    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-credential-socket\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.635686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635563    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-token\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-token\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.635686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635603    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-kube-api-access-xmgp9\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-kube-api-access-xmgp9\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.635686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635639    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-socket\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.635927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635687    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-certs\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.635927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635736    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-podinfo\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.635927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635810    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-envoy\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.635927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635730    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-data" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:19:02.635927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635849    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-credential-socket" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:19:02.635927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635885    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istiod-ca-cert\") pod \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\" (UID: \"d95d2c4d-0f27-452d-bf5c-687dd8b95243\") "
Nov 01 15:19:02.636243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.635988    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-socket" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:19:02.636243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.636062    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-certs" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:19:02.636361 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.636339    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istiod-ca-cert" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:19:02.636486 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.636467    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:02.636564 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.636487    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:02.636564 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.636501    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:02.636564 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.636514    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:02.636564 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.636528    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:02.637916 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.637889    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-token" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:19:02.638041 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.637982    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-kube-api-access-xmgp9" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "kube-api-access-xmgp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:19:02.638269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.638243    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-envoy" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:19:02.638269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.638253    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-podinfo" pod "d95d2c4d-0f27-452d-bf5c-687dd8b95243" (UID: "d95d2c4d-0f27-452d-bf5c-687dd8b95243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:19:02.737445 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.737409    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:02.737445 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.737446    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmgp9\" (UniqueName: \"kubernetes.io/projected/d95d2c4d-0f27-452d-bf5c-687dd8b95243-kube-api-access-xmgp9\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:02.737676 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.737458    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:02.737676 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:02.737468    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95d2c4d-0f27-452d-bf5c-687dd8b95243-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:03.071041 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.071010    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"]
Nov 01 15:19:03.071679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.071662    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy"
Nov 01 15:19:03.071746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.071680    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy"
Nov 01 15:19:03.071746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.071693    5423 cpu_manager.go:414] "containerMap: removing container" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy"
Nov 01 15:19:03.071746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.071698    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy"
Nov 01 15:19:03.071849 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.071786    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy"
Nov 01 15:19:03.071849 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.071793    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerName="istio-proxy"
Nov 01 15:19:03.075707 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.075685    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.129484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.129452    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dda3bf67b7e4\""
Nov 01 15:19:03.131974 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.131946    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"]
Nov 01 15:19:03.142050 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.142015    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-routing-overlay-volume\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.142197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.142068    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hn6\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-kube-api-access-b9hn6\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.142285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.142195    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-provider-credentials\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.142285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.142255    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-plugins-config-volume\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.243447 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.243380    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-routing-overlay-volume\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.243636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.243491    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hn6\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-kube-api-access-b9hn6\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.243636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.243589    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-provider-credentials\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.243773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.243637    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-plugins-config-volume\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.244174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.244149    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-routing-overlay-volume\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.244353 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.244335    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-plugins-config-volume\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.246215 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.246196    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-provider-credentials\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.302076 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.302049    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hn6\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-kube-api-access-b9hn6\") pod \"payload-processing-external-model-7cdcb77d89-4nxlw\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") " pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.328776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.328707    5423 generic.go:356] "Generic (PLEG): container finished" podID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" containerID="7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc" exitCode=0
Nov 01 15:19:03.328776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.328756    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"
Nov 01 15:19:03.328988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.328798    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" event={"ID":"d95d2c4d-0f27-452d-bf5c-687dd8b95243","Type":"ContainerDied","Data":"7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc"}
Nov 01 15:19:03.328988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.328863    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj" event={"ID":"d95d2c4d-0f27-452d-bf5c-687dd8b95243","Type":"ContainerDied","Data":"27e11d45aab34bfe4c4f2cd4d38035a9c2b4d1175fbe484ac09d649f4b684acd"}
Nov 01 15:19:03.328988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.328894    5423 scope.go:122] "RemoveContainer" containerID="7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc"
Nov 01 15:19:03.338947 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.338927    5423 scope.go:122] "RemoveContainer" containerID="83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd"
Nov 01 15:19:03.347444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.347427    5423 scope.go:122] "RemoveContainer" containerID="7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc"
Nov 01 15:19:03.347698 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:19:03.347679    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc\": container with ID starting with 7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc not found: ID does not exist" containerID="7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc"
Nov 01 15:19:03.347748 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.347704    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc"} err="failed to get container status \"7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc\": rpc error: code = NotFound desc = could not find container \"7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc\": container with ID starting with 7d43129f6c4d1680d216bccfe4466aab88a1190ded2c7f8f8002e48f210d87dc not found: ID does not exist"
Nov 01 15:19:03.347748 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.347725    5423 scope.go:122] "RemoveContainer" containerID="83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd"
Nov 01 15:19:03.347949 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:19:03.347931    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd\": container with ID starting with 83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd not found: ID does not exist" containerID="83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd"
Nov 01 15:19:03.347994 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.347952    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd"} err="failed to get container status \"83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd\": rpc error: code = NotFound desc = could not find container \"83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd\": container with ID starting with 83f2f617b5c445a1a185230f5b8ee376f47ec12fffd1aa743ab06387a23821cd not found: ID does not exist"
Nov 01 15:19:03.390652 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.390625    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:03.498071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.498041    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"]
Nov 01 15:19:03.501803 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.501774    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e92165-openshift-default-79bc7b77b5-5rlvj"]
Nov 01 15:19:03.763657 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:19:03.763624    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79dab53c_2958_4d8b_92f6_a4c7061cc741.slice/crio-3c83960ed4a04f5dd25fdb22db4f45b7d7176dcc20c4bd5ca54dad8cd1b5445b WatchSource:0}: Error finding container 3c83960ed4a04f5dd25fdb22db4f45b7d7176dcc20c4bd5ca54dad8cd1b5445b: Status 404 returned error can't find the container with id 3c83960ed4a04f5dd25fdb22db4f45b7d7176dcc20c4bd5ca54dad8cd1b5445b
Nov 01 15:19:03.766636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.766606    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"]
Nov 01 15:19:03.960828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:03.960792    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95d2c4d-0f27-452d-bf5c-687dd8b95243" path="/var/lib/kubelet/pods/d95d2c4d-0f27-452d-bf5c-687dd8b95243/volumes"
Nov 01 15:19:04.335522 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:04.335477    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw" event={"ID":"79dab53c-2958-4d8b-92f6-a4c7061cc741","Type":"ContainerStarted","Data":"ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9"}
Nov 01 15:19:04.335734 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:04.335543    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw" event={"ID":"79dab53c-2958-4d8b-92f6-a4c7061cc741","Type":"ContainerStarted","Data":"3c83960ed4a04f5dd25fdb22db4f45b7d7176dcc20c4bd5ca54dad8cd1b5445b"}
Nov 01 15:19:04.335734 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:04.335639    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:04.452256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:04.452200    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw" podStartSLOduration=1.452185196 podStartE2EDuration="1.452185196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:04.451470039 +0000 UTC m=+3684.616955336" watchObservedRunningTime="2026-11-01 15:19:04.452185196 +0000 UTC m=+3684.617670535"
Nov 01 15:19:05.149236 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:19:05.149184    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:19:07.042525 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:07.042488    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"]
Nov 01 15:19:07.151326 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:07.151288    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"]
Nov 01 15:19:12.045493 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.045446    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc" podUID="5ee82c73-c5e8-4d23-baba-761ae54fd9e7" containerName="payload-processing" containerID="cri-o://a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c" gracePeriod=25
Nov 01 15:19:12.151962 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.151914    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98" podUID="b5e8e262-4059-44bd-a845-83ffbebadb69" containerName="payload-pre-processing" containerID="cri-o://aed7b36814db15874f3ce48a2b810d7c4bf3ea8b2dd3961baa34d27d52f9efb8" gracePeriod=25
Nov 01 15:19:12.197802 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:19:12.197766    5423 logging.go:55] [core] [Channel #8156 SubChannel #8157]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.195:50052", ServerName: "10.133.0.195:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.195:50052: connect: connection refused"
Nov 01 15:19:12.198321 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:19:12.198298    5423 logging.go:55] [core] [Channel #8158 SubChannel #8159]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.194:50052", ServerName: "10.133.0.194:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.194:50052: connect: connection refused"
Nov 01 15:19:12.350923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.350888    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:19:12.419847 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.419808    5423 generic.go:356] "Generic (PLEG): container finished" podID="b5e8e262-4059-44bd-a845-83ffbebadb69" containerID="aed7b36814db15874f3ce48a2b810d7c4bf3ea8b2dd3961baa34d27d52f9efb8" exitCode=0
Nov 01 15:19:12.420016 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.419879    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98" event={"ID":"b5e8e262-4059-44bd-a845-83ffbebadb69","Type":"ContainerDied","Data":"aed7b36814db15874f3ce48a2b810d7c4bf3ea8b2dd3961baa34d27d52f9efb8"}
Nov 01 15:19:12.421647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.421622    5423 generic.go:356] "Generic (PLEG): container finished" podID="5ee82c73-c5e8-4d23-baba-761ae54fd9e7" containerID="a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c" exitCode=0
Nov 01 15:19:12.421787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.421688    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc" event={"ID":"5ee82c73-c5e8-4d23-baba-761ae54fd9e7","Type":"ContainerDied","Data":"a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c"}
Nov 01 15:19:12.421787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.421701    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"
Nov 01 15:19:12.421787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.421738    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc" event={"ID":"5ee82c73-c5e8-4d23-baba-761ae54fd9e7","Type":"ContainerDied","Data":"4b42f55f1a10181893353a238fb947894310d13717e6ffa9d7b017338305a846"}
Nov 01 15:19:12.421787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.421767    5423 scope.go:122] "RemoveContainer" containerID="a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c"
Nov 01 15:19:12.434827 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.434812    5423 scope.go:122] "RemoveContainer" containerID="a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c"
Nov 01 15:19:12.435140 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:19:12.435120    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c\": container with ID starting with a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c not found: ID does not exist" containerID="a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c"
Nov 01 15:19:12.435219 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.435151    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c"} err="failed to get container status \"a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c\": rpc error: code = NotFound desc = could not find container \"a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c\": container with ID starting with a1640ca55bacc337424e6f8cfa1b9a29bf1d81b75ed98c5075d6c5cbf457791c not found: ID does not exist"
Nov 01 15:19:12.469698 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.469677    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:19:12.471004 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.470972    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-plugins-config-volume\") pod \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\" (UID: \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\") "
Nov 01 15:19:12.471160 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.471143    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-kube-api-access-g99z2\" (UniqueName: \"kubernetes.io/projected/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-kube-api-access-g99z2\") pod \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\" (UID: \"5ee82c73-c5e8-4d23-baba-761ae54fd9e7\") "
Nov 01 15:19:12.471318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.471295    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-plugins-config-volume" pod "5ee82c73-c5e8-4d23-baba-761ae54fd9e7" (UID: "5ee82c73-c5e8-4d23-baba-761ae54fd9e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:19:12.471581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.471563    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:12.473189 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.473160    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-kube-api-access-g99z2" pod "5ee82c73-c5e8-4d23-baba-761ae54fd9e7" (UID: "5ee82c73-c5e8-4d23-baba-761ae54fd9e7"). InnerVolumeSpecName "kube-api-access-g99z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:19:12.572210 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.572180    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e8e262-4059-44bd-a845-83ffbebadb69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e8e262-4059-44bd-a845-83ffbebadb69-plugins-config-volume\") pod \"b5e8e262-4059-44bd-a845-83ffbebadb69\" (UID: \"b5e8e262-4059-44bd-a845-83ffbebadb69\") "
Nov 01 15:19:12.572419 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.572242    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e8e262-4059-44bd-a845-83ffbebadb69-kube-api-access-9g7db\" (UniqueName: \"kubernetes.io/projected/b5e8e262-4059-44bd-a845-83ffbebadb69-kube-api-access-9g7db\") pod \"b5e8e262-4059-44bd-a845-83ffbebadb69\" (UID: \"b5e8e262-4059-44bd-a845-83ffbebadb69\") "
Nov 01 15:19:12.572588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.572575    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g99z2\" (UniqueName: \"kubernetes.io/projected/5ee82c73-c5e8-4d23-baba-761ae54fd9e7-kube-api-access-g99z2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:12.572663 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.572625    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e8e262-4059-44bd-a845-83ffbebadb69-plugins-config-volume" pod "b5e8e262-4059-44bd-a845-83ffbebadb69" (UID: "b5e8e262-4059-44bd-a845-83ffbebadb69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:19:12.574606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.574584    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e8e262-4059-44bd-a845-83ffbebadb69-kube-api-access-9g7db" pod "b5e8e262-4059-44bd-a845-83ffbebadb69" (UID: "b5e8e262-4059-44bd-a845-83ffbebadb69"). InnerVolumeSpecName "kube-api-access-9g7db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:19:12.673272 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.673241    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e8e262-4059-44bd-a845-83ffbebadb69-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:12.673272 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.673269    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7db\" (UniqueName: \"kubernetes.io/projected/b5e8e262-4059-44bd-a845-83ffbebadb69-kube-api-access-9g7db\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:12.844910 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.844828    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"]
Nov 01 15:19:12.848452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:12.848428    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc"]
Nov 01 15:19:13.197573 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.197497    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-928a13-b4f47bc97-wvwcc" podUID="5ee82c73-c5e8-4d23-baba-761ae54fd9e7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.195:50052\" within 1s: context deadline exceeded"
Nov 01 15:19:13.198554 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.198528    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98" podUID="b5e8e262-4059-44bd-a845-83ffbebadb69" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.194:50052\" within 1s: context deadline exceeded"
Nov 01 15:19:13.429310 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.429164    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"
Nov 01 15:19:13.429558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.429174    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98" event={"ID":"b5e8e262-4059-44bd-a845-83ffbebadb69","Type":"ContainerDied","Data":"d95a9e093d024019ea0c5ff108ae302e3ce338f420287e9bbbb56618b0fa821b"}
Nov 01 15:19:13.429558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.429451    5423 scope.go:122] "RemoveContainer" containerID="aed7b36814db15874f3ce48a2b810d7c4bf3ea8b2dd3961baa34d27d52f9efb8"
Nov 01 15:19:13.603759 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.603731    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"]
Nov 01 15:19:13.606599 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.606568    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-928a13-69c9f6c959-2vc98"]
Nov 01 15:19:13.961229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.961148    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee82c73-c5e8-4d23-baba-761ae54fd9e7" path="/var/lib/kubelet/pods/5ee82c73-c5e8-4d23-baba-761ae54fd9e7/volumes"
Nov 01 15:19:13.961610 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:13.961596    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e8e262-4059-44bd-a845-83ffbebadb69" path="/var/lib/kubelet/pods/b5e8e262-4059-44bd-a845-83ffbebadb69/volumes"
Nov 01 15:19:15.345359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:15.345330    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:19:15.515819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:15.515788    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"]
Nov 01 15:19:20.517020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.516940    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w" podUID="6d37e2d1-e62a-402f-a538-2a05845a2076" containerName="payload-processing" containerID="cri-o://a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa" gracePeriod=25
Nov 01 15:19:20.630967 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.630936    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"]
Nov 01 15:19:20.631565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.631550    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b5e8e262-4059-44bd-a845-83ffbebadb69" containerName="payload-pre-processing"
Nov 01 15:19:20.631610 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.631566    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e8e262-4059-44bd-a845-83ffbebadb69" containerName="payload-pre-processing"
Nov 01 15:19:20.631610 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.631593    5423 cpu_manager.go:414] "containerMap: removing container" podUID="5ee82c73-c5e8-4d23-baba-761ae54fd9e7" containerName="payload-processing"
Nov 01 15:19:20.631610 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.631598    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee82c73-c5e8-4d23-baba-761ae54fd9e7" containerName="payload-processing"
Nov 01 15:19:20.631726 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.631718    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee82c73-c5e8-4d23-baba-761ae54fd9e7" containerName="payload-processing"
Nov 01 15:19:20.631761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.631733    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e8e262-4059-44bd-a845-83ffbebadb69" containerName="payload-pre-processing"
Nov 01 15:19:20.635683 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.635598    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.681451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.681414    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"]
Nov 01 15:19:20.685567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.685546    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.737090 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.737063    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"]
Nov 01 15:19:20.764803 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.764757    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.764996 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.764827    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.764996 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.764964    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkfnj\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-kube-api-access-fkfnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.765128 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.765008    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.765128 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.765068    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.765247 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.765159    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.765247 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.765192    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.765356 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.765319    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.765356 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.765348    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.789330 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.789301    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"]
Nov 01 15:19:20.814725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.814696    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:19:20.866155 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866129    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-plugins-config-volume\") pod \"6d37e2d1-e62a-402f-a538-2a05845a2076\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") "
Nov 01 15:19:20.866327 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866236    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-kube-api-access-mf9f7\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-kube-api-access-mf9f7\") pod \"6d37e2d1-e62a-402f-a538-2a05845a2076\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") "
Nov 01 15:19:20.866418 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866367    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-provider-credentials\") pod \"6d37e2d1-e62a-402f-a538-2a05845a2076\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") "
Nov 01 15:19:20.866489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866470    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-routing-overlay-volume\") pod \"6d37e2d1-e62a-402f-a538-2a05845a2076\" (UID: \"6d37e2d1-e62a-402f-a538-2a05845a2076\") "
Nov 01 15:19:20.866680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866639    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.866680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866690    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.866911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866729    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.866911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866762    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.866911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866801    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.866911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866832    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.866911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866880    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkfnj\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-kube-api-access-fkfnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.867511 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866914    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.867907 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.866640    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-plugins-config-volume" pod "6d37e2d1-e62a-402f-a538-2a05845a2076" (UID: "6d37e2d1-e62a-402f-a538-2a05845a2076"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:19:20.868653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.868616    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-routing-overlay-volume" pod "6d37e2d1-e62a-402f-a538-2a05845a2076" (UID: "6d37e2d1-e62a-402f-a538-2a05845a2076"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:19:20.869201 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869157    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-kube-api-access-mf9f7" pod "6d37e2d1-e62a-402f-a538-2a05845a2076" (UID: "6d37e2d1-e62a-402f-a538-2a05845a2076"). InnerVolumeSpecName "kube-api-access-mf9f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:19:20.869377 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869349    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.869495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869426    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.869495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869445    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.869679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869498    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.869679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869530    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.869679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869582    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.869679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869612    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.869679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869664    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zftk\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-kube-api-access-8zftk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.870053 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.869717    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.870053 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870020    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-provider-credentials" pod "6d37e2d1-e62a-402f-a538-2a05845a2076" (UID: "6d37e2d1-e62a-402f-a538-2a05845a2076"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:19:20.871017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870327    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.871017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870377    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.871017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870516    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.871017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870624    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.871017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870764    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:20.871017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870782    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:20.871017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870800    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d37e2d1-e62a-402f-a538-2a05845a2076-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:20.871017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.870817    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf9f7\" (UniqueName: \"kubernetes.io/projected/6d37e2d1-e62a-402f-a538-2a05845a2076-kube-api-access-mf9f7\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:19:20.871541 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.871032    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.872812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.872757    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.873126 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.873108    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.873958 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.873936    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.926778 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.926747    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkfnj\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-kube-api-access-fkfnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.927938 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.927920    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4dgqf\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.950938 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.950907    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:20.971773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.971745    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.971891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.971791    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.971891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.971824    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.971891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.971855    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.971944    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.971979    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.972026    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zftk\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-kube-api-access-8zftk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.972076    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.972115    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.972164    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.972200    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972920 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.972811    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.972920 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.972824    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.973219 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.973051    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.975559 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.974552    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:20.975877 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:20.975847    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:21.030991 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.030949    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zftk\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-kube-api-access-8zftk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:21.031145 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.031124    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8zq9l\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:21.311072 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.311042    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:21.383488 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.382939    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"]
Nov 01 15:19:21.383488 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:19:21.383433    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cbaac4_0a96_4e67_989a_d3a216ef06d6.slice/crio-6bc440573106eb27634b7238c2f60bf3487914869d3273783973b3bc2ea1edea WatchSource:0}: Error finding container 6bc440573106eb27634b7238c2f60bf3487914869d3273783973b3bc2ea1edea: Status 404 returned error can't find the container with id 6bc440573106eb27634b7238c2f60bf3487914869d3273783973b3bc2ea1edea
Nov 01 15:19:21.386671 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.386633    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:21.386791 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.386744    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:21.386860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.386795    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:21.519239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.510174    5423 generic.go:356] "Generic (PLEG): container finished" podID="6d37e2d1-e62a-402f-a538-2a05845a2076" containerID="a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa" exitCode=0
Nov 01 15:19:21.519239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.510264    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w" event={"ID":"6d37e2d1-e62a-402f-a538-2a05845a2076","Type":"ContainerDied","Data":"a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa"}
Nov 01 15:19:21.519239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.510310    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w" event={"ID":"6d37e2d1-e62a-402f-a538-2a05845a2076","Type":"ContainerDied","Data":"5a596a5938227de87c6bd481245bafe45a27d81c211af3b0f9bcd90a0c89049d"}
Nov 01 15:19:21.519239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.510337    5423 scope.go:122] "RemoveContainer" containerID="a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa"
Nov 01 15:19:21.519239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.510508    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"
Nov 01 15:19:21.521248 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.521211    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" event={"ID":"66cbaac4-0a96-4e67-989a-d3a216ef06d6","Type":"ContainerStarted","Data":"6bc440573106eb27634b7238c2f60bf3487914869d3273783973b3bc2ea1edea"}
Nov 01 15:19:21.529876 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.529853    5423 scope.go:122] "RemoveContainer" containerID="a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa"
Nov 01 15:19:21.530111 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:19:21.530089    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa\": container with ID starting with a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa not found: ID does not exist" containerID="a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa"
Nov 01 15:19:21.530176 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.530120    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa"} err="failed to get container status \"a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa\": rpc error: code = NotFound desc = could not find container \"a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa\": container with ID starting with a52d145c8a68ccfdd738f7ed5b0074d481e095bbdb855c82e5a384f99400c3fa not found: ID does not exist"
Nov 01 15:19:21.686849 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.686771    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"]
Nov 01 15:19:21.693196 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.693167    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"]
Nov 01 15:19:21.700953 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.700921    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb7f88854-2s59w"]
Nov 01 15:19:21.728308 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:19:21.728281    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffdd6ed_54e1_4ec2_8773_dc8f9d46edc8.slice/crio-1cd28fd4948a3df36c37624d40e48e3ca1293abe4d8fdea5fc09464f0134448e WatchSource:0}: Error finding container 1cd28fd4948a3df36c37624d40e48e3ca1293abe4d8fdea5fc09464f0134448e: Status 404 returned error can't find the container with id 1cd28fd4948a3df36c37624d40e48e3ca1293abe4d8fdea5fc09464f0134448e
Nov 01 15:19:21.731403 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.731358    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:21.731502 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.731470    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:21.731578 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.731502    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:21.738196 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.738165    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"]
Nov 01 15:19:21.738876 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.738848    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6d37e2d1-e62a-402f-a538-2a05845a2076" containerName="payload-processing"
Nov 01 15:19:21.738876 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.738868    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d37e2d1-e62a-402f-a538-2a05845a2076" containerName="payload-processing"
Nov 01 15:19:21.739067 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.739055    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d37e2d1-e62a-402f-a538-2a05845a2076" containerName="payload-processing"
Nov 01 15:19:21.747500 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.747475    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.765635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.765599    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"]
Nov 01 15:19:21.860177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.859842    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-af2dd2-openshift-default-dockercfg-llvkx\""
Nov 01 15:19:21.905284 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.905243    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520751d7-df0b-4875-8e79-9d9703298a93-istiod-ca-cert\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.905486 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.905420    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520751d7-df0b-4875-8e79-9d9703298a93-istio-podinfo\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.905563 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.905534    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-certs\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.905845 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.905795    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-istio-token\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.905953 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.905926    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-credential-socket\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.906023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.905965    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtd2\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-kube-api-access-8qtd2\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.906102 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.906062    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-envoy\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.906181 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.906160    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-data\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.906242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.906229    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-socket\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:21.963051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:21.962961    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d37e2d1-e62a-402f-a538-2a05845a2076" path="/var/lib/kubelet/pods/6d37e2d1-e62a-402f-a538-2a05845a2076/volumes"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008126    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520751d7-df0b-4875-8e79-9d9703298a93-istiod-ca-cert\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008212    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520751d7-df0b-4875-8e79-9d9703298a93-istio-podinfo\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008282    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-certs\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008328    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-istio-token\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008409    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-credential-socket\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008440    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtd2\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-kube-api-access-8qtd2\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008474    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-envoy\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008529    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-data\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.008574    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-socket\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.009097    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-socket\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.009736    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520751d7-df0b-4875-8e79-9d9703298a93-istiod-ca-cert\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.010411    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-credential-socket\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.011408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.010654    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-certs\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.017969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.017939    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-envoy\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.019017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.018712    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-data\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.025969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.025939    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520751d7-df0b-4875-8e79-9d9703298a93-istio-podinfo\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.068757 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.068725    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-istio-token\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.076085 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.076042    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtd2\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-kube-api-access-8qtd2\") pod \"e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") " pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.088937 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.088906    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:22.548374 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.548340    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"]
Nov 01 15:19:22.553608 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:19:22.553580    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520751d7_df0b_4875_8e79_9d9703298a93.slice/crio-79d21d91b991d14df30719b961d78790360023ace6045b761cdc6ce91fc7bd21 WatchSource:0}: Error finding container 79d21d91b991d14df30719b961d78790360023ace6045b761cdc6ce91fc7bd21: Status 404 returned error can't find the container with id 79d21d91b991d14df30719b961d78790360023ace6045b761cdc6ce91fc7bd21
Nov 01 15:19:22.556802 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.556763    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:22.556880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.556862    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:22.556927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.556909    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:19:22.560417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.560367    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" event={"ID":"66cbaac4-0a96-4e67-989a-d3a216ef06d6","Type":"ContainerStarted","Data":"634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e"}
Nov 01 15:19:22.562058 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.562036    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" event={"ID":"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8","Type":"ContainerStarted","Data":"99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24"}
Nov 01 15:19:22.562156 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.562070    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" event={"ID":"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8","Type":"ContainerStarted","Data":"1cd28fd4948a3df36c37624d40e48e3ca1293abe4d8fdea5fc09464f0134448e"}
Nov 01 15:19:22.682431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.682356    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podStartSLOduration=2.682341567 podStartE2EDuration="2.682341567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:22.680850777 +0000 UTC m=+3702.846336080" watchObservedRunningTime="2026-11-01 15:19:22.682341567 +0000 UTC m=+3702.847826865"
Nov 01 15:19:22.835411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.826965    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podStartSLOduration=2.826944309 podStartE2EDuration="2.826944309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:22.826756333 +0000 UTC m=+3702.992241654" watchObservedRunningTime="2026-11-01 15:19:22.826944309 +0000 UTC m=+3702.992429609"
Nov 01 15:19:22.959409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.959159    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:22.960787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.959945    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 01 15:19:22.960787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:22.959997    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 01 15:19:23.326410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:23.316533    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:23.326410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:23.316663    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 15:19:23.326410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:23.316709    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 15:19:23.570933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:23.570893    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" event={"ID":"520751d7-df0b-4875-8e79-9d9703298a93","Type":"ContainerStarted","Data":"7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29"}
Nov 01 15:19:23.571350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:23.570950    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" event={"ID":"520751d7-df0b-4875-8e79-9d9703298a93","Type":"ContainerStarted","Data":"79d21d91b991d14df30719b961d78790360023ace6045b761cdc6ce91fc7bd21"}
Nov 01 15:19:23.698459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:23.697711    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" podStartSLOduration=2.697692444 podStartE2EDuration="2.697692444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:23.695860203 +0000 UTC m=+3703.861345502" watchObservedRunningTime="2026-11-01 15:19:23.697692444 +0000 UTC m=+3703.863177745"
Nov 01 15:19:23.951496 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:23.951274    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 01 15:19:23.951496 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:23.951339    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 01 15:19:24.089314 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:24.089278    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:24.102409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:24.094661    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:24.313418 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:24.313367    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 15:19:24.313695 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:24.313662    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 15:19:24.578918 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:24.578837    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:24.579684 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:24.579662    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:19:24.951943 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:24.951855    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 01 15:19:24.951943 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:24.951909    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 01 15:19:25.320408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:25.318703    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 15:19:25.320408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:25.318768    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 15:19:25.951915 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:25.951880    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 01 15:19:25.952334 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:25.951931    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 01 15:19:26.316407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:26.312954    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 15:19:26.316407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:26.313026    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 15:19:26.951749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:26.951714    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 01 15:19:26.951945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:26.951763    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 01 15:19:27.315466 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:27.311879    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 15:19:27.315466 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:27.311958    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 15:19:27.952165 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:27.952125    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 01 15:19:27.952605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:27.952190    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 01 15:19:28.311483 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:28.311444    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 15:19:28.311644 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:28.311515    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 15:19:28.951521 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:28.951475    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 01 15:19:28.951689 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:28.951545    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 01 15:19:29.311666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:29.311629    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 01 15:19:29.312177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:29.311687    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 01 15:19:30.579563 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:30.579517    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:30.582219 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:30.582194    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:30.583191 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:30.583168    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:19:30.925363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:30.925273    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:30.926296 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:30.926272    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:30.927050 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:30.927031    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:19:35.149095 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:19:35.149060    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:19:40.314552 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:19:40.314516    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:20:05.149565 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:05.149521    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:20:09.435991 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.435944    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"]
Nov 01 15:20:09.436538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.436227    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95" podUID="cae598df-f01c-4c4b-81d2-adb23a5ee6f2" containerName="maas-api" containerID="cri-o://02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277" gracePeriod=30
Nov 01 15:20:09.761568 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.761543    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:20:09.787135 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.787106    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-openshift-service-ca\") pod \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") "
Nov 01 15:20:09.787291 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.787155    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-kube-api-access-cq2m7\" (UniqueName: \"kubernetes.io/projected/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-kube-api-access-cq2m7\") pod \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") "
Nov 01 15:20:09.787291 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.787235    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls\") pod \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") "
Nov 01 15:20:09.787291 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.787287    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-metrics-certs\") pod \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\" (UID: \"cae598df-f01c-4c4b-81d2-adb23a5ee6f2\") "
Nov 01 15:20:09.787482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.787459    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-openshift-service-ca" pod "cae598df-f01c-4c4b-81d2-adb23a5ee6f2" (UID: "cae598df-f01c-4c4b-81d2-adb23a5ee6f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:20:09.787806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.787784    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:09.789350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.789323    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls" pod "cae598df-f01c-4c4b-81d2-adb23a5ee6f2" (UID: "cae598df-f01c-4c4b-81d2-adb23a5ee6f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:20:09.789486 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.789431    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-kube-api-access-cq2m7" pod "cae598df-f01c-4c4b-81d2-adb23a5ee6f2" (UID: "cae598df-f01c-4c4b-81d2-adb23a5ee6f2"). InnerVolumeSpecName "kube-api-access-cq2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:20:09.789620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.789602    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-metrics-certs" pod "cae598df-f01c-4c4b-81d2-adb23a5ee6f2" (UID: "cae598df-f01c-4c4b-81d2-adb23a5ee6f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:20:09.915422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.894967    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:09.915422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.895009    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:09.915422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.895025    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq2m7\" (UniqueName: \"kubernetes.io/projected/cae598df-f01c-4c4b-81d2-adb23a5ee6f2-kube-api-access-cq2m7\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:09.944856 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.944819    5423 generic.go:356] "Generic (PLEG): container finished" podID="cae598df-f01c-4c4b-81d2-adb23a5ee6f2" containerID="02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277" exitCode=0
Nov 01 15:20:09.945052 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.944928    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"
Nov 01 15:20:09.945052 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.944956    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95" event={"ID":"cae598df-f01c-4c4b-81d2-adb23a5ee6f2","Type":"ContainerDied","Data":"02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277"}
Nov 01 15:20:09.945052 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.945017    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95" event={"ID":"cae598df-f01c-4c4b-81d2-adb23a5ee6f2","Type":"ContainerDied","Data":"9457ae9ac4350563b4ccc33d9f65fdc8c8f4c4219bbce0d646b251fb0643250c"}
Nov 01 15:20:09.945052 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.945048    5423 scope.go:122] "RemoveContainer" containerID="02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277"
Nov 01 15:20:09.965734 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.965715    5423 scope.go:122] "RemoveContainer" containerID="02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277"
Nov 01 15:20:09.965994 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:09.965976    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277\": container with ID starting with 02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277 not found: ID does not exist" containerID="02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277"
Nov 01 15:20:09.966043 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:09.966002    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277"} err="failed to get container status \"02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277\": rpc error: code = NotFound desc = could not find container \"02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277\": container with ID starting with 02777271ad7b80036db9fd0a9ad96520597edb6c8fe5b63edc3f3b98cdcde277 not found: ID does not exist"
Nov 01 15:20:10.125493 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:10.125463    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"]
Nov 01 15:20:10.131365 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:10.131339    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-928a13-fc89df44c-7hd95"]
Nov 01 15:20:11.824622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.816107    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"]
Nov 01 15:20:11.824622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.817594    5423 cpu_manager.go:414] "containerMap: removing container" podUID="cae598df-f01c-4c4b-81d2-adb23a5ee6f2" containerName="maas-api"
Nov 01 15:20:11.824622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.817615    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae598df-f01c-4c4b-81d2-adb23a5ee6f2" containerName="maas-api"
Nov 01 15:20:11.824622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.817967    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae598df-f01c-4c4b-81d2-adb23a5ee6f2" containerName="maas-api"
Nov 01 15:20:11.824622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.823966    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:11.847353 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.847314    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"]
Nov 01 15:20:11.877767 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.877733    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-287c3706f19e\""
Nov 01 15:20:11.929701 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.929665    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-plugins-config-volume\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:11.929701 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.929716    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqm7s\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-kube-api-access-rqm7s\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:11.929942 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.929856    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:11.929990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.929964    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-routing-overlay-volume\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:11.960819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:11.960790    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae598df-f01c-4c4b-81d2-adb23a5ee6f2" path="/var/lib/kubelet/pods/cae598df-f01c-4c4b-81d2-adb23a5ee6f2/volumes"
Nov 01 15:20:12.031089 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.031055    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.031305 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.031139    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-routing-overlay-volume\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.031426 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.031399    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-plugins-config-volume\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.031501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.031451    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqm7s\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-kube-api-access-rqm7s\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.031893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.031860    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-routing-overlay-volume\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.032200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.032177    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-plugins-config-volume\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.033846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.033826    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.089103 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.089029    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqm7s\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-kube-api-access-rqm7s\") pod \"payload-processing-external-model-5f9f8d5b5-mnrd5\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") " pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.137812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.137780    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:12.511833 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:20:12.511793    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2e9bc3_d6ee_4b1b_a980_0db37b000cab.slice/crio-b6544511032ee7255cb1f560bea993afb6f5bcbf6653f771fb5df520be45f3a7 WatchSource:0}: Error finding container b6544511032ee7255cb1f560bea993afb6f5bcbf6653f771fb5df520be45f3a7: Status 404 returned error can't find the container with id b6544511032ee7255cb1f560bea993afb6f5bcbf6653f771fb5df520be45f3a7
Nov 01 15:20:12.514532 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.514507    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"]
Nov 01 15:20:12.966019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.965986    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5" event={"ID":"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab","Type":"ContainerStarted","Data":"d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4"}
Nov 01 15:20:12.966441 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.966037    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5" event={"ID":"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab","Type":"ContainerStarted","Data":"b6544511032ee7255cb1f560bea993afb6f5bcbf6653f771fb5df520be45f3a7"}
Nov 01 15:20:12.966441 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:12.966071    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:13.099421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:13.099346    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5" podStartSLOduration=2.099332839 podStartE2EDuration="2.099332839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:13.098445779 +0000 UTC m=+3753.263931076" watchObservedRunningTime="2026-11-01 15:20:13.099332839 +0000 UTC m=+3753.264818137"
Nov 01 15:20:23.975002 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:23.974967    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:20:24.146160 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.146130    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"]
Nov 01 15:20:24.153171 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:24.153139    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:24.153171 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:24.153169    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:24.153440 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:24.153231    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:20:24.653216257 +0000 UTC m=+3764.818701532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:24.221130 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.221099    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"]
Nov 01 15:20:24.232941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.232868    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.282037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.282004    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"]
Nov 01 15:20:24.355643 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.355609    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqmr\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-kube-api-access-gpqmr\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.355643 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.355645    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-plugins-config-volume\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.355946 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.355921    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-provider-credentials\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.355999 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.355981    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-routing-overlay-volume\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.457177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.457148    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqmr\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-kube-api-access-gpqmr\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.457177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.457182    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-plugins-config-volume\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.457381 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.457317    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-provider-credentials\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.457381 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.457344    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-routing-overlay-volume\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.458027 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.458001    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-routing-overlay-volume\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.458027 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.458017    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-plugins-config-volume\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.459885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.459869    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-provider-credentials\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.515304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.515232    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqmr\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-kube-api-access-gpqmr\") pod \"payload-processing-external-model-5bf79b58f7-lvczg\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.552323 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.552291    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:24.698930 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:24.666765    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:24.698930 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:24.666803    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:24.698930 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:24.666880    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:20:25.666860972 +0000 UTC m=+3765.832346253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:24.933705 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:20:24.933672    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363651e2_d919_4ea7_8190_79b590d2e993.slice/crio-c9acd848e0e91057a72e691c0946273007c37ffd2482e1139cdfaa494be6e0ef WatchSource:0}: Error finding container c9acd848e0e91057a72e691c0946273007c37ffd2482e1139cdfaa494be6e0ef: Status 404 returned error can't find the container with id c9acd848e0e91057a72e691c0946273007c37ffd2482e1139cdfaa494be6e0ef
Nov 01 15:20:24.937259 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:24.937235    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"]
Nov 01 15:20:25.053755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:25.053711    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg" event={"ID":"363651e2-d919-4ea7-8190-79b590d2e993","Type":"ContainerStarted","Data":"b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823"}
Nov 01 15:20:25.054113 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:25.053777    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg" event={"ID":"363651e2-d919-4ea7-8190-79b590d2e993","Type":"ContainerStarted","Data":"c9acd848e0e91057a72e691c0946273007c37ffd2482e1139cdfaa494be6e0ef"}
Nov 01 15:20:25.054113 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:25.053904    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:25.171844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:25.171793    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg" podStartSLOduration=1.171780472 podStartE2EDuration="1.171780472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:25.170819691 +0000 UTC m=+3765.336304988" watchObservedRunningTime="2026-11-01 15:20:25.171780472 +0000 UTC m=+3765.337265770"
Nov 01 15:20:25.691967 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:25.691929    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:25.691967 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:25.691958    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:25.692201 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:25.692011    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:20:27.691997214 +0000 UTC m=+3767.857482494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:26.740402 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:26.740357    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"]
Nov 01 15:20:27.721902 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:27.721866    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:27.721902 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:27.721895    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:27.722180 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:27.721969    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:20:31.721950413 +0000 UTC m=+3771.887435688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:29.147464 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.147425    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw" podUID="79dab53c-2958-4d8b-92f6-a4c7061cc741" containerName="payload-processing" containerID="cri-o://ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9" gracePeriod=25
Nov 01 15:20:29.406376 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.406314    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:20:29.545317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.545281    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-kube-api-access-b9hn6\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-kube-api-access-b9hn6\") pod \"79dab53c-2958-4d8b-92f6-a4c7061cc741\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") "
Nov 01 15:20:29.545317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.545317    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-provider-credentials\") pod \"79dab53c-2958-4d8b-92f6-a4c7061cc741\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") "
Nov 01 15:20:29.545632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.545366    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-plugins-config-volume\") pod \"79dab53c-2958-4d8b-92f6-a4c7061cc741\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") "
Nov 01 15:20:29.545632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.545591    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-routing-overlay-volume\") pod \"79dab53c-2958-4d8b-92f6-a4c7061cc741\" (UID: \"79dab53c-2958-4d8b-92f6-a4c7061cc741\") "
Nov 01 15:20:29.546073 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.545871    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-plugins-config-volume" pod "79dab53c-2958-4d8b-92f6-a4c7061cc741" (UID: "79dab53c-2958-4d8b-92f6-a4c7061cc741"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:20:29.546073 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.546041    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-routing-overlay-volume" pod "79dab53c-2958-4d8b-92f6-a4c7061cc741" (UID: "79dab53c-2958-4d8b-92f6-a4c7061cc741"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:20:29.546294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.546272    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:29.546362 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.546302    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79dab53c-2958-4d8b-92f6-a4c7061cc741-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:29.547784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.547666    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-provider-credentials" pod "79dab53c-2958-4d8b-92f6-a4c7061cc741" (UID: "79dab53c-2958-4d8b-92f6-a4c7061cc741"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:20:29.547784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.547753    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-kube-api-access-b9hn6" pod "79dab53c-2958-4d8b-92f6-a4c7061cc741" (UID: "79dab53c-2958-4d8b-92f6-a4c7061cc741"). InnerVolumeSpecName "kube-api-access-b9hn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:20:29.648203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.648165    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9hn6\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-kube-api-access-b9hn6\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:29.648203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:29.648203    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79dab53c-2958-4d8b-92f6-a4c7061cc741-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:30.086801 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.086767    5423 generic.go:356] "Generic (PLEG): container finished" podID="79dab53c-2958-4d8b-92f6-a4c7061cc741" containerID="ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9" exitCode=0
Nov 01 15:20:30.087057 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.086812    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"
Nov 01 15:20:30.087057 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.086852    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw" event={"ID":"79dab53c-2958-4d8b-92f6-a4c7061cc741","Type":"ContainerDied","Data":"ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9"}
Nov 01 15:20:30.087057 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.086919    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw" event={"ID":"79dab53c-2958-4d8b-92f6-a4c7061cc741","Type":"ContainerDied","Data":"3c83960ed4a04f5dd25fdb22db4f45b7d7176dcc20c4bd5ca54dad8cd1b5445b"}
Nov 01 15:20:30.087057 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.086952    5423 scope.go:122] "RemoveContainer" containerID="ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9"
Nov 01 15:20:30.096523 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.096504    5423 scope.go:122] "RemoveContainer" containerID="ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9"
Nov 01 15:20:30.096783 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:30.096765    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9\": container with ID starting with ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9 not found: ID does not exist" containerID="ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9"
Nov 01 15:20:30.096833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.096791    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9"} err="failed to get container status \"ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9\": rpc error: code = NotFound desc = could not find container \"ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9\": container with ID starting with ede8fa7df7d63c4b3b2ed2b9cf8528b1f31baf61af017c10d40eafd9d9453ba9 not found: ID does not exist"
Nov 01 15:20:30.260362 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.260270    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"]
Nov 01 15:20:30.288425 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:30.288372    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cdcb77d89-4nxlw"]
Nov 01 15:20:31.794155 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:31.794124    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:31.794155 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:31.794148    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:31.794698 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:31.794243    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:20:39.794224688 +0000 UTC m=+3779.959709978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:32.026593 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.026550    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79dab53c-2958-4d8b-92f6-a4c7061cc741" path="/var/lib/kubelet/pods/79dab53c-2958-4d8b-92f6-a4c7061cc741/volumes"
Nov 01 15:20:32.066786 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.066751    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg" podUID="363651e2-d919-4ea7-8190-79b590d2e993" containerName="payload-processing" containerID="cri-o://b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823" gracePeriod=25
Nov 01 15:20:32.260120 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.260087    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"]
Nov 01 15:20:32.261150 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.261126    5423 cpu_manager.go:414] "containerMap: removing container" podUID="79dab53c-2958-4d8b-92f6-a4c7061cc741" containerName="payload-processing"
Nov 01 15:20:32.261288 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.261278    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79dab53c-2958-4d8b-92f6-a4c7061cc741" containerName="payload-processing"
Nov 01 15:20:32.261513 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.261501    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="79dab53c-2958-4d8b-92f6-a4c7061cc741" containerName="payload-processing"
Nov 01 15:20:32.265062 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.265046    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:32.303359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.303322    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlzhv\" (UniqueName: \"kubernetes.io/projected/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-kube-api-access-dlzhv\") pod \"payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m\" (UID: \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:32.303521 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.303487    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m\" (UID: \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:32.325026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.323484    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fc8c64-dockercfg-n59s9\""
Nov 01 15:20:32.325026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.324093    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fc8c64\""
Nov 01 15:20:32.369165 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.369126    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"]
Nov 01 15:20:32.441784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.440065    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlzhv\" (UniqueName: \"kubernetes.io/projected/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-kube-api-access-dlzhv\") pod \"payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m\" (UID: \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:32.441784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.440172    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m\" (UID: \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:32.441784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.441193    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m\" (UID: \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:32.483124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.483098    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:32.506714 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.506680    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlzhv\" (UniqueName: \"kubernetes.io/projected/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-kube-api-access-dlzhv\") pod \"payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m\" (UID: \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:32.541430 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.541376    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-plugins-config-volume\") pod \"363651e2-d919-4ea7-8190-79b590d2e993\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") "
Nov 01 15:20:32.541586 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.541446    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-provider-credentials\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-provider-credentials\") pod \"363651e2-d919-4ea7-8190-79b590d2e993\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") "
Nov 01 15:20:32.541586 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.541567    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-routing-overlay-volume\") pod \"363651e2-d919-4ea7-8190-79b590d2e993\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") "
Nov 01 15:20:32.541721 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.541652    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-kube-api-access-gpqmr\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-kube-api-access-gpqmr\") pod \"363651e2-d919-4ea7-8190-79b590d2e993\" (UID: \"363651e2-d919-4ea7-8190-79b590d2e993\") "
Nov 01 15:20:32.541868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.541843    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-plugins-config-volume" pod "363651e2-d919-4ea7-8190-79b590d2e993" (UID: "363651e2-d919-4ea7-8190-79b590d2e993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:20:32.541973 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.541950    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-routing-overlay-volume" pod "363651e2-d919-4ea7-8190-79b590d2e993" (UID: "363651e2-d919-4ea7-8190-79b590d2e993"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:20:32.542173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.542154    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:32.542221 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.542182    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/363651e2-d919-4ea7-8190-79b590d2e993-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:32.543679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.543655    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-provider-credentials" pod "363651e2-d919-4ea7-8190-79b590d2e993" (UID: "363651e2-d919-4ea7-8190-79b590d2e993"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:20:32.544010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.543982    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-kube-api-access-gpqmr" pod "363651e2-d919-4ea7-8190-79b590d2e993" (UID: "363651e2-d919-4ea7-8190-79b590d2e993"). InnerVolumeSpecName "kube-api-access-gpqmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:20:32.580829 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.580781    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:32.643991 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.643961    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpqmr\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-kube-api-access-gpqmr\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:32.644152 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.644135    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/363651e2-d919-4ea7-8190-79b590d2e993-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:32.952041 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:20:32.952004    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd83a1fb_1f07_4fb6_9d44_2f2316f4399d.slice/crio-53476323bf6451e6ffc7ac0f90d2c784749443675a70f9b2aa3a01749cb77887 WatchSource:0}: Error finding container 53476323bf6451e6ffc7ac0f90d2c784749443675a70f9b2aa3a01749cb77887: Status 404 returned error can't find the container with id 53476323bf6451e6ffc7ac0f90d2c784749443675a70f9b2aa3a01749cb77887
Nov 01 15:20:32.955044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:32.955023    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"]
Nov 01 15:20:33.116785 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.116746    5423 generic.go:356] "Generic (PLEG): container finished" podID="363651e2-d919-4ea7-8190-79b590d2e993" containerID="b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823" exitCode=0
Nov 01 15:20:33.116976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.116794    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"
Nov 01 15:20:33.116976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.116828    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg" event={"ID":"363651e2-d919-4ea7-8190-79b590d2e993","Type":"ContainerDied","Data":"b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823"}
Nov 01 15:20:33.116976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.116890    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg" event={"ID":"363651e2-d919-4ea7-8190-79b590d2e993","Type":"ContainerDied","Data":"c9acd848e0e91057a72e691c0946273007c37ffd2482e1139cdfaa494be6e0ef"}
Nov 01 15:20:33.116976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.116923    5423 scope.go:122] "RemoveContainer" containerID="b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823"
Nov 01 15:20:33.118786 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.118760    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m" event={"ID":"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d","Type":"ContainerStarted","Data":"12f847325d3b8692816ee8de867f348d51ca8780caf8998abfdca6cbf459da62"}
Nov 01 15:20:33.118945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.118807    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m" event={"ID":"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d","Type":"ContainerStarted","Data":"53476323bf6451e6ffc7ac0f90d2c784749443675a70f9b2aa3a01749cb77887"}
Nov 01 15:20:33.118945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.118850    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:33.131097 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.131075    5423 scope.go:122] "RemoveContainer" containerID="b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823"
Nov 01 15:20:33.131409 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:33.131376    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823\": container with ID starting with b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823 not found: ID does not exist" containerID="b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823"
Nov 01 15:20:33.131495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.131420    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823"} err="failed to get container status \"b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823\": rpc error: code = NotFound desc = could not find container \"b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823\": container with ID starting with b4792d2612aa5110984d2f6fd1d7be623aecb55dcd6114028ed0aaa64db84823 not found: ID does not exist"
Nov 01 15:20:33.289333 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.289241    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m" podStartSLOduration=1.289228375 podStartE2EDuration="1.289228375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:33.287134108 +0000 UTC m=+3773.452619405" watchObservedRunningTime="2026-11-01 15:20:33.289228375 +0000 UTC m=+3773.454713672"
Nov 01 15:20:33.423415 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.420936    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"]
Nov 01 15:20:33.425879 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.425658    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-lvczg"]
Nov 01 15:20:33.962090 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:33.962055    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363651e2-d919-4ea7-8190-79b590d2e993" path="/var/lib/kubelet/pods/363651e2-d919-4ea7-8190-79b590d2e993/volumes"
Nov 01 15:20:34.113514 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.113475    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"]
Nov 01 15:20:34.114323 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.114305    5423 cpu_manager.go:414] "containerMap: removing container" podUID="363651e2-d919-4ea7-8190-79b590d2e993" containerName="payload-processing"
Nov 01 15:20:34.114417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.114327    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363651e2-d919-4ea7-8190-79b590d2e993" containerName="payload-processing"
Nov 01 15:20:34.114555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.114542    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="363651e2-d919-4ea7-8190-79b590d2e993" containerName="payload-processing"
Nov 01 15:20:34.119850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.119824    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.175097 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.175067    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"]
Nov 01 15:20:34.225885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.225809    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"]
Nov 01 15:20:34.229982 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.229960    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.266516 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.266479    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-plugins-config-volume\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.266687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.266549    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-provider-credentials\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.266762 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.266699    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scbd\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-kube-api-access-7scbd\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.266835 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.266794    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-routing-overlay-volume\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.287458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.287429    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"]
Nov 01 15:20:34.299893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.299865    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3d5a5b-openshift-default-dockercfg-9rjhp\""
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.368609    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-provider-credentials\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.368710    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-envoy\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.368752    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scbd\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-kube-api-access-7scbd\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.368786    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-credential-socket\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.368813    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-data\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.368863    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31553b-417a-48a7-b424-5e4d0135ca4b-istiod-ca-cert\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.368896    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-routing-overlay-volume\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.368952    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-certs\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.369033    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-token\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.369073    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-kube-api-access-5v9bk\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.369154    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-podinfo\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.369279    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-socket\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.369329    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-plugins-config-volume\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.373260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.373194    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-routing-overlay-volume\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.403420 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.375948    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-plugins-config-volume\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.445419 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.444959    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-provider-credentials\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473191    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-envoy\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473264    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-credential-socket\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473289    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-data\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473345    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31553b-417a-48a7-b424-5e4d0135ca4b-istiod-ca-cert\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473457    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-certs\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473541    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-token\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473576    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-kube-api-access-5v9bk\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473654    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-podinfo\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.473699    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-socket\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.474455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.474220    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-socket\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.483246 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.482567    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-envoy\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.483881 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.483844    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-certs\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.484345 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.484290    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-credential-socket\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.484832 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.484812    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-data\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.485611 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.485590    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31553b-417a-48a7-b424-5e4d0135ca4b-istiod-ca-cert\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.494027 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.490989    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-podinfo\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.513965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.513189    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scbd\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-kube-api-access-7scbd\") pod \"payload-processing-external-model-5bf79b58f7-78pfr\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") " pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.544231 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.544199    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-token\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.544408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.544356    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-kube-api-access-5v9bk\") pod \"e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") " pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:34.735510 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.735121    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:34.843128 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:34.843092    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:35.112760 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:20:35.112724    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629214ee_6040_4e09_bfdc_842234ff1d26.slice/crio-f00fae8203abe1e2980833801727d3e093f4049cef100666e4d09cc20c8724b9 WatchSource:0}: Error finding container f00fae8203abe1e2980833801727d3e093f4049cef100666e4d09cc20c8724b9: Status 404 returned error can't find the container with id f00fae8203abe1e2980833801727d3e093f4049cef100666e4d09cc20c8724b9
Nov 01 15:20:35.116699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:35.116674    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"]
Nov 01 15:20:35.135192 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:35.135164    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" event={"ID":"629214ee-6040-4e09-bfdc-842234ff1d26","Type":"ContainerStarted","Data":"f00fae8203abe1e2980833801727d3e093f4049cef100666e4d09cc20c8724b9"}
Nov 01 15:20:35.153646 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:35.153617    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:20:35.256773 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:20:35.256745    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa31553b_417a_48a7_b424_5e4d0135ca4b.slice/crio-40afc59161168b1673603b96c4a75024e30b9b047adafda50f7d231599b5fd91 WatchSource:0}: Error finding container 40afc59161168b1673603b96c4a75024e30b9b047adafda50f7d231599b5fd91: Status 404 returned error can't find the container with id 40afc59161168b1673603b96c4a75024e30b9b047adafda50f7d231599b5fd91
Nov 01 15:20:35.259479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:35.259449    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:20:35.259565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:35.259511    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:20:35.259565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:35.259539    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:20:35.260684 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:35.260663    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"]
Nov 01 15:20:36.143612 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:36.143577    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" event={"ID":"629214ee-6040-4e09-bfdc-842234ff1d26","Type":"ContainerStarted","Data":"0b0dbf6d51e27f3eacf19b4a5a107d555755126cfa84f2e7509a45f354c198b6"}
Nov 01 15:20:36.144052 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:36.143648    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:20:36.145316 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:36.145293    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" event={"ID":"aa31553b-417a-48a7-b424-5e4d0135ca4b","Type":"ContainerStarted","Data":"82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54"}
Nov 01 15:20:36.145441 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:36.145333    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" event={"ID":"aa31553b-417a-48a7-b424-5e4d0135ca4b","Type":"ContainerStarted","Data":"40afc59161168b1673603b96c4a75024e30b9b047adafda50f7d231599b5fd91"}
Nov 01 15:20:36.260675 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:36.260622    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podStartSLOduration=2.260609882 podStartE2EDuration="2.260609882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:36.259187075 +0000 UTC m=+3776.424672373" watchObservedRunningTime="2026-11-01 15:20:36.260609882 +0000 UTC m=+3776.426095179"
Nov 01 15:20:36.378590 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:36.378534    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" podStartSLOduration=2.378521242 podStartE2EDuration="2.378521242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:36.377737903 +0000 UTC m=+3776.543223201" watchObservedRunningTime="2026-11-01 15:20:36.378521242 +0000 UTC m=+3776.544006540"
Nov 01 15:20:36.843936 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:36.843903    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:36.850061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:36.850036    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:37.153797 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:37.153716    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:37.154754 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:37.154733    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:20:37.181014 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:37.180979    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:20:39.801412 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:39.799631    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:39.801412 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:39.799664    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:39.801412 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:39.799724    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:20:55.799705189 +0000 UTC m=+3795.965190479 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:44.130932 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:44.130898    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:20:47.156736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:47.156692    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:20:55.815138 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:55.815059    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:55.815138 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:55.815087    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:55.815138 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:20:55.815135    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:21:27.815122833 +0000 UTC m=+3827.980608121 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : secret "agc-cred-eb52a593c198" not found
Nov 01 15:20:57.155302 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:20:57.155266    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:21:05.151026 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:21:05.150983    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:21:07.154605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:07.154569    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:21:17.154844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:17.154807    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:21:22.605411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:22.604436    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 01 15:21:22.650409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:22.649527    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:21:22.837564 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:22.837519    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:21:27.154792 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:27.154746    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:21:27.881210 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:21:27.881175    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:21:27.881210 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:21:27.881204    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: secret "agc-cred-eb52a593c198" not found
Nov 01 15:21:27.881472 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:21:27.881264    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:22:31.881249551 +0000 UTC m=+3892.046734827 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : secret "agc-cred-eb52a593c198" not found
Nov 01 15:21:29.670523 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:29.670493    5423 generic.go:356] "Generic (PLEG): container finished" podID="520751d7-df0b-4875-8e79-9d9703298a93" containerID="7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29" exitCode=0
Nov 01 15:21:29.671139 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:29.670534    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" event={"ID":"520751d7-df0b-4875-8e79-9d9703298a93","Type":"ContainerDied","Data":"7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29"}
Nov 01 15:21:29.671139 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:29.670870    5423 scope.go:122] "RemoveContainer" containerID="7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29"
Nov 01 15:21:29.672567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:29.672537    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:21:29.672654 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:29.672607    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:21:29.672654 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:29.672638    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:21:30.678873 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:30.678838    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" event={"ID":"520751d7-df0b-4875-8e79-9d9703298a93","Type":"ContainerStarted","Data":"8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694"}
Nov 01 15:21:31.090015 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:31.089978    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:21:31.090318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:31.090293    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 15:21:31.090475 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:31.090347    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 15:21:32.089574 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:32.089540    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 15:21:32.089988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:32.089592    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 15:21:33.089481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:33.089441    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 15:21:33.089669 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:33.089500    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 15:21:34.089661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:34.089624    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 15:21:34.090071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:34.089681    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 15:21:35.089199 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:35.089167    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 15:21:35.089376 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:35.089212    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 15:21:35.150549 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:21:35.150512    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:21:36.089448 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:36.089378    5423 patch_prober.go:29] interesting pod/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 01 15:21:36.089681 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:36.089515    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 01 15:21:36.203490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:36.203451    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:21:37.094103 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:37.094071    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:21:37.094573 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:37.094556    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:21:37.095143 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:37.095124    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:21:37.155340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:37.155308    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:21:47.154886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:47.154848    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:21:57.178555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:21:57.178507    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:22:00.925548 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:00.925500    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:22:00.926173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:00.926139    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:22:02.954470 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:02.954433    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:22:05.150582 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:22:05.150533    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:22:12.954489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:12.954449    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:22:22.954667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:22.954587    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:22:23.014993 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.014955    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"]
Nov 01 15:22:23.016123 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.016096    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:22:23.065866 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.065839    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"]
Nov 01 15:22:23.070024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.070007    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.130525 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.130493    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"]
Nov 01 15:22:23.227047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.226957    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-routing-overlay-volume\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.227215 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.227190    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-plugins-config-volume\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.227346 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.227326    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4l7g\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-kube-api-access-v4l7g\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.227434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.227361    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-provider-credentials\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.329032 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.328990    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4l7g\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-kube-api-access-v4l7g\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.329032 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.329033    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-provider-credentials\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.329283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.329097    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-routing-overlay-volume\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.329283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.329201    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-plugins-config-volume\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.329885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.329859    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-routing-overlay-volume\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.330023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.329924    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-plugins-config-volume\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.331605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.331582    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-provider-credentials\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.418781 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.418746    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4l7g\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-kube-api-access-v4l7g\") pod \"payload-processing-external-model-7b6df859f6-69h8t\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") " pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:23.684295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:23.684253    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:24.098860 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:22:24.098821    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291332d8_abef_4e91_b966_149b19fd1939.slice/crio-d07494af7312b5c0bba3ab71ab432cde6d93fe709a66bee83a33b4ca6eff4875 WatchSource:0}: Error finding container d07494af7312b5c0bba3ab71ab432cde6d93fe709a66bee83a33b4ca6eff4875: Status 404 returned error can't find the container with id d07494af7312b5c0bba3ab71ab432cde6d93fe709a66bee83a33b4ca6eff4875
Nov 01 15:22:24.103164 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.103141    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"]
Nov 01 15:22:24.146789 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.146759    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" event={"ID":"291332d8-abef-4e91-b966-149b19fd1939","Type":"ContainerStarted","Data":"d07494af7312b5c0bba3ab71ab432cde6d93fe709a66bee83a33b4ca6eff4875"}
Nov 01 15:22:24.458159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.458080    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"]
Nov 01 15:22:24.462128 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.462111    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.546528 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.544176    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"]
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576162    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istiod-ca-cert\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576216    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-data\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576297    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-envoy\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576366    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89dv\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-kube-api-access-x89dv\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576464    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-podinfo\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576500    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-token\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576538    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-credential-socket\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576566    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-certs\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.596421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.576671    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-socket\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.619408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.614796    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-775095-openshift-default-dockercfg-pfqdn\""
Nov 01 15:22:24.677605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.677573    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istiod-ca-cert\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.677787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.677610    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-data\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.677787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.677661    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-envoy\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.677787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.677701    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89dv\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-kube-api-access-x89dv\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.677960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.677862    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-podinfo\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.677960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.677919    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-token\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.678083 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.677962    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-credential-socket\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.678083 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.677989    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-certs\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.678211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.678077    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-data\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.678211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.678107    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-socket\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.678492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.678464    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istiod-ca-cert\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.678599 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.678494    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-credential-socket\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.678599 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.678515    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-certs\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.678743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.678717    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-socket\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.680481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.680461    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-podinfo\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.680700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.680679    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-envoy\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.736530 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.736460    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-token\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.736642 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.736583    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89dv\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-kube-api-access-x89dv\") pod \"e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") " pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:24.777449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:24.777421    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:25.154664 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:25.154588    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" event={"ID":"291332d8-abef-4e91-b966-149b19fd1939","Type":"ContainerStarted","Data":"20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d"}
Nov 01 15:22:25.154664 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:25.154664    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:22:25.204595 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:22:25.204568    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6427e8b0_629c_444c_b8bc_bc9aa51cc50d.slice/crio-869babec676f31545038f956e8acf89c8e9ad940792f075e05d9e91f481d46a6 WatchSource:0}: Error finding container 869babec676f31545038f956e8acf89c8e9ad940792f075e05d9e91f481d46a6: Status 404 returned error can't find the container with id 869babec676f31545038f956e8acf89c8e9ad940792f075e05d9e91f481d46a6
Nov 01 15:22:25.207188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:25.207152    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:22:25.207315 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:25.207225    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:22:25.207315 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:25.207254    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:22:25.211295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:25.211273    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"]
Nov 01 15:22:25.272492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:25.272442    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podStartSLOduration=2.272428593 podStartE2EDuration="2.272428593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:22:25.270610695 +0000 UTC m=+3885.436095995" watchObservedRunningTime="2026-11-01 15:22:25.272428593 +0000 UTC m=+3885.437913891"
Nov 01 15:22:26.162443 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:26.162403    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" event={"ID":"6427e8b0-629c-444c-b8bc-bc9aa51cc50d","Type":"ContainerStarted","Data":"6f5112d87654345d95788afede0733d5d34e98c0cbbcde45a3f12ff19b6e7907"}
Nov 01 15:22:26.162833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:26.162460    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" event={"ID":"6427e8b0-629c-444c-b8bc-bc9aa51cc50d","Type":"ContainerStarted","Data":"869babec676f31545038f956e8acf89c8e9ad940792f075e05d9e91f481d46a6"}
Nov 01 15:22:26.335401 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:26.335333    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" podStartSLOduration=2.335319364 podStartE2EDuration="2.335319364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:22:26.332517868 +0000 UTC m=+3886.498003165" watchObservedRunningTime="2026-11-01 15:22:26.335319364 +0000 UTC m=+3886.500804663"
Nov 01 15:22:26.778124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:26.778089    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:26.784187 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:26.784163    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:27.168399 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:27.168362    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:27.169359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:27.169340    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:22:28.015500 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.015441    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing" containerID="cri-o://0b0dbf6d51e27f3eacf19b4a5a107d555755126cfa84f2e7509a45f354c198b6" gracePeriod=25
Nov 01 15:22:28.176368 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.176334    5423 generic.go:356] "Generic (PLEG): container finished" podID="629214ee-6040-4e09-bfdc-842234ff1d26" containerID="0b0dbf6d51e27f3eacf19b4a5a107d555755126cfa84f2e7509a45f354c198b6" exitCode=0
Nov 01 15:22:28.176784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.176373    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" event={"ID":"629214ee-6040-4e09-bfdc-842234ff1d26","Type":"ContainerDied","Data":"0b0dbf6d51e27f3eacf19b4a5a107d555755126cfa84f2e7509a45f354c198b6"}
Nov 01 15:22:28.284856 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.284803    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:22:28.451488 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.451454    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-plugins-config-volume\") pod \"629214ee-6040-4e09-bfdc-842234ff1d26\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") "
Nov 01 15:22:28.451691 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.451579    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-kube-api-access-7scbd\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-kube-api-access-7scbd\") pod \"629214ee-6040-4e09-bfdc-842234ff1d26\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") "
Nov 01 15:22:28.451762 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.451745    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-routing-overlay-volume\") pod \"629214ee-6040-4e09-bfdc-842234ff1d26\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") "
Nov 01 15:22:28.451874 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.451850    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-plugins-config-volume" pod "629214ee-6040-4e09-bfdc-842234ff1d26" (UID: "629214ee-6040-4e09-bfdc-842234ff1d26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:22:28.451942 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.451863    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-provider-credentials\") pod \"629214ee-6040-4e09-bfdc-842234ff1d26\" (UID: \"629214ee-6040-4e09-bfdc-842234ff1d26\") "
Nov 01 15:22:28.452138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.452110    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-routing-overlay-volume" pod "629214ee-6040-4e09-bfdc-842234ff1d26" (UID: "629214ee-6040-4e09-bfdc-842234ff1d26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:22:28.452623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.452562    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:22:28.452623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.452589    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629214ee-6040-4e09-bfdc-842234ff1d26-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:22:28.453869 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.453849    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-kube-api-access-7scbd" pod "629214ee-6040-4e09-bfdc-842234ff1d26" (UID: "629214ee-6040-4e09-bfdc-842234ff1d26"). InnerVolumeSpecName "kube-api-access-7scbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:22:28.453953 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.453921    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-provider-credentials" pod "629214ee-6040-4e09-bfdc-842234ff1d26" (UID: "629214ee-6040-4e09-bfdc-842234ff1d26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:22:28.585411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.576845    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:22:28.585411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:28.576887    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7scbd\" (UniqueName: \"kubernetes.io/projected/629214ee-6040-4e09-bfdc-842234ff1d26-kube-api-access-7scbd\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:22:29.183821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:29.183786    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"
Nov 01 15:22:29.184359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:29.183825    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr" event={"ID":"629214ee-6040-4e09-bfdc-842234ff1d26","Type":"ContainerDied","Data":"f00fae8203abe1e2980833801727d3e093f4049cef100666e4d09cc20c8724b9"}
Nov 01 15:22:29.184359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:29.183887    5423 scope.go:122] "RemoveContainer" containerID="0b0dbf6d51e27f3eacf19b4a5a107d555755126cfa84f2e7509a45f354c198b6"
Nov 01 15:22:29.354859 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:29.354827    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"]
Nov 01 15:22:29.405932 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:29.405897    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bf79b58f7-78pfr"]
Nov 01 15:22:29.961319 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:29.961281    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" path="/var/lib/kubelet/pods/629214ee-6040-4e09-bfdc-842234ff1d26/volumes"
Nov 01 15:22:31.917808 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:22:31.917775    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-dda3bf67b7e4: secret "agc-cred-dda3bf67b7e4" not found
Nov 01 15:22:31.917808 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:22:31.917802    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:22:31.917808 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:22:31.917813    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: [secret "agc-cred-dda3bf67b7e4" not found, secret "agc-cred-eb52a593c198" not found]
Nov 01 15:22:31.918290 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:22:31.917865    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:24:33.917851095 +0000 UTC m=+4014.083336385 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : [secret "agc-cred-dda3bf67b7e4" not found, secret "agc-cred-eb52a593c198" not found]
Nov 01 15:22:35.149537 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:22:35.149496    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:22:36.165758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:36.165716    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:22:40.214883 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:40.214846    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:22:42.924407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:42.924069    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:22:43.030799 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:43.030764    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:22:46.202609 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.202531    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:22:46.463567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.461030    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"]
Nov 01 15:22:46.463567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.462061    5423 cpu_manager.go:414] "containerMap: removing container" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing"
Nov 01 15:22:46.463567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.462085    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing"
Nov 01 15:22:46.463567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.462257    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="629214ee-6040-4e09-bfdc-842234ff1d26" containerName="payload-processing"
Nov 01 15:22:46.474207 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.472017    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:46.523333 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.523301    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"]
Nov 01 15:22:46.531519 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.531486    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a68c75-a162-42fa-a0f9-486c3dbe91d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v\" (UID: \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:46.531682 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.531641    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4kp\" (UniqueName: \"kubernetes.io/projected/75a68c75-a162-42fa-a0f9-486c3dbe91d9-kube-api-access-6l4kp\") pod \"payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v\" (UID: \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:46.574214 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.574155    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5feb76\""
Nov 01 15:22:46.575157 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.575134    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5feb76-dockercfg-pfvjl\""
Nov 01 15:22:46.662559 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.662502    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"]
Nov 01 15:22:46.672685 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.672656    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:46.711028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.710999    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a68c75-a162-42fa-a0f9-486c3dbe91d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v\" (UID: \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:46.712538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.712303    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a68c75-a162-42fa-a0f9-486c3dbe91d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v\" (UID: \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:46.754462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.745229    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"]
Nov 01 15:22:46.754462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.745380    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4kp\" (UniqueName: \"kubernetes.io/projected/75a68c75-a162-42fa-a0f9-486c3dbe91d9-kube-api-access-6l4kp\") pod \"payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v\" (UID: \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:46.827858 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.827145    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4kp\" (UniqueName: \"kubernetes.io/projected/75a68c75-a162-42fa-a0f9-486c3dbe91d9-kube-api-access-6l4kp\") pod \"payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v\" (UID: \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:46.847160 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.847132    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290225-35e5-412e-9bc4-3f9767008bb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx\" (UID: \"9c290225-35e5-412e-9bc4-3f9767008bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:46.847310 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.847174    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6tq\" (UniqueName: \"kubernetes.io/projected/9c290225-35e5-412e-9bc4-3f9767008bb4-kube-api-access-sj6tq\") pod \"payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx\" (UID: \"9c290225-35e5-412e-9bc4-3f9767008bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:46.948435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.948375    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290225-35e5-412e-9bc4-3f9767008bb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx\" (UID: \"9c290225-35e5-412e-9bc4-3f9767008bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:46.948622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.948454    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6tq\" (UniqueName: \"kubernetes.io/projected/9c290225-35e5-412e-9bc4-3f9767008bb4-kube-api-access-sj6tq\") pod \"payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx\" (UID: \"9c290225-35e5-412e-9bc4-3f9767008bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:46.949092 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:46.949067    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290225-35e5-412e-9bc4-3f9767008bb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx\" (UID: \"9c290225-35e5-412e-9bc4-3f9767008bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:47.015645 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:47.008941    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6tq\" (UniqueName: \"kubernetes.io/projected/9c290225-35e5-412e-9bc4-3f9767008bb4-kube-api-access-sj6tq\") pod \"payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx\" (UID: \"9c290225-35e5-412e-9bc4-3f9767008bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:47.024403 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:47.023601    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:47.121429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:47.089028    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:47.416034 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:22:47.415995    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c290225_35e5_412e_9bc4_3f9767008bb4.slice/crio-97ce2c8acc2ce71fbd8101cca22649ec8504aac94bdcaaad7acf7f03d14b0c75 WatchSource:0}: Error finding container 97ce2c8acc2ce71fbd8101cca22649ec8504aac94bdcaaad7acf7f03d14b0c75: Status 404 returned error can't find the container with id 97ce2c8acc2ce71fbd8101cca22649ec8504aac94bdcaaad7acf7f03d14b0c75
Nov 01 15:22:47.421346 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:47.421209    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"]
Nov 01 15:22:47.509937 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:22:47.509885    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a68c75_a162_42fa_a0f9_486c3dbe91d9.slice/crio-d4711089a65a4d1f732a1e947fdda7d598c9efe80a3ad20100a8ac5b78b5a976 WatchSource:0}: Error finding container d4711089a65a4d1f732a1e947fdda7d598c9efe80a3ad20100a8ac5b78b5a976: Status 404 returned error can't find the container with id d4711089a65a4d1f732a1e947fdda7d598c9efe80a3ad20100a8ac5b78b5a976
Nov 01 15:22:47.514601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:47.514579    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"]
Nov 01 15:22:48.416648 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:48.416596    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v" event={"ID":"75a68c75-a162-42fa-a0f9-486c3dbe91d9","Type":"ContainerStarted","Data":"8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880"}
Nov 01 15:22:48.417163 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:48.416669    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:48.417163 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:48.416733    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v" event={"ID":"75a68c75-a162-42fa-a0f9-486c3dbe91d9","Type":"ContainerStarted","Data":"d4711089a65a4d1f732a1e947fdda7d598c9efe80a3ad20100a8ac5b78b5a976"}
Nov 01 15:22:48.418652 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:48.418625    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx" event={"ID":"9c290225-35e5-412e-9bc4-3f9767008bb4","Type":"ContainerStarted","Data":"b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1"}
Nov 01 15:22:48.418793 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:48.418674    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx" event={"ID":"9c290225-35e5-412e-9bc4-3f9767008bb4","Type":"ContainerStarted","Data":"97ce2c8acc2ce71fbd8101cca22649ec8504aac94bdcaaad7acf7f03d14b0c75"}
Nov 01 15:22:48.418793 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:48.418751    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:22:48.541784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:48.541719    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v" podStartSLOduration=2.541693104 podStartE2EDuration="2.541693104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:22:48.535491849 +0000 UTC m=+3908.700977147" watchObservedRunningTime="2026-11-01 15:22:48.541693104 +0000 UTC m=+3908.707178403"
Nov 01 15:22:48.672575 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:48.672450    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx" podStartSLOduration=2.672427087 podStartE2EDuration="2.672427087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:22:48.671299329 +0000 UTC m=+3908.836784641" watchObservedRunningTime="2026-11-01 15:22:48.672427087 +0000 UTC m=+3908.837912385"
Nov 01 15:22:51.442112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:51.442081    5423 generic.go:356] "Generic (PLEG): container finished" podID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerID="82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54" exitCode=0
Nov 01 15:22:51.442585 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:51.442128    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" event={"ID":"aa31553b-417a-48a7-b424-5e4d0135ca4b","Type":"ContainerDied","Data":"82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54"}
Nov 01 15:22:51.442585 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:51.442518    5423 scope.go:122] "RemoveContainer" containerID="82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54"
Nov 01 15:22:51.444130 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:51.444095    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:22:51.444264 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:51.444182    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:22:51.444264 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:51.444231    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:22:52.153494 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:52.153458    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:22:52.450925 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:52.450834    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" event={"ID":"aa31553b-417a-48a7-b424-5e4d0135ca4b","Type":"ContainerStarted","Data":"8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53"}
Nov 01 15:22:52.843906 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:52.843874    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:22:52.845406 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:52.845356    5423 patch_prober.go:29] interesting pod/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 01 15:22:52.845525 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:52.845418    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 01 15:22:53.843830 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:53.843794    5423 patch_prober.go:29] interesting pod/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 01 15:22:53.844362 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:53.843878    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 01 15:22:54.844355 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:54.844322    5423 patch_prober.go:29] interesting pod/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 01 15:22:54.844808 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:54.844378    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 01 15:22:55.843820 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:55.843783    5423 patch_prober.go:29] interesting pod/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 01 15:22:55.844030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:55.843847    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 01 15:22:56.164450 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:56.164359    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:22:56.844474 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:56.844440    5423 patch_prober.go:29] interesting pod/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 01 15:22:56.844678 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:56.844498    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 01 15:22:57.848124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:57.848091    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:22:57.848653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:57.848633    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:22:57.849310 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:57.849289    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:22:59.431609 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:59.431577    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:22:59.432050 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:22:59.431647    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:23:05.149526 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:23:05.149487    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:23:06.164713 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:06.164677    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:23:11.484021 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.483986    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"]
Nov 01 15:23:11.488417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.488380    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.558027 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.557996    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-89e5ef-openshift-default-dockercfg-dlvc9\""
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669417    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-certs\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669617    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9892c3e6-7a37-4d91-b593-a71576e0fd52-istiod-ca-cert\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669666    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-token\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669723    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-data\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669753    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-podinfo\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669814    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-envoy\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669855    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-credential-socket\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669883    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-socket\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.673410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.669921    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpzg5\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-kube-api-access-cpzg5\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.769642 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.769535    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"]
Nov 01 15:23:11.770961 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.770933    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9892c3e6-7a37-4d91-b593-a71576e0fd52-istiod-ca-cert\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.771474 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.771437    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-token\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.771693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.771672    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-data\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.771817 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.771802    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-podinfo\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.771971 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.771950    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-envoy\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.772112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.772096    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-credential-socket\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.772234 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.772221    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-socket\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.772354 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.772341    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpzg5\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-kube-api-access-cpzg5\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.772543 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.772526    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-certs\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.773183 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.773161    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-certs\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.773771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.773748    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-credential-socket\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.773964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.773939    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-socket\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.774627 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.774584    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-data\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.775588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.775566    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9892c3e6-7a37-4d91-b593-a71576e0fd52-istiod-ca-cert\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.777623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.777580    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-podinfo\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.777760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.777744    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-envoy\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.851931 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.851865    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-token\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:11.855281 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:11.855249    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpzg5\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-kube-api-access-cpzg5\") pod \"e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") " pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:12.106946 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:12.106911    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:12.530864 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:12.530792    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"]
Nov 01 15:23:12.561924 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:23:12.561897    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9892c3e6_7a37_4d91_b593_a71576e0fd52.slice/crio-20276e3e008c6347e8ecb2363413f7434939d76809c7188b27b4c32b9eae5ed0 WatchSource:0}: Error finding container 20276e3e008c6347e8ecb2363413f7434939d76809c7188b27b4c32b9eae5ed0: Status 404 returned error can't find the container with id 20276e3e008c6347e8ecb2363413f7434939d76809c7188b27b4c32b9eae5ed0
Nov 01 15:23:12.564595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:12.564564    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:23:12.564687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:12.564638    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:23:12.564687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:12.564673    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:23:12.777117 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:12.777069    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" event={"ID":"9892c3e6-7a37-4d91-b593-a71576e0fd52","Type":"ContainerStarted","Data":"75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c"}
Nov 01 15:23:12.777308 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:12.777167    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" event={"ID":"9892c3e6-7a37-4d91-b593-a71576e0fd52","Type":"ContainerStarted","Data":"20276e3e008c6347e8ecb2363413f7434939d76809c7188b27b4c32b9eae5ed0"}
Nov 01 15:23:12.897781 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:12.897733    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" podStartSLOduration=1.897719785 podStartE2EDuration="1.897719785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:23:12.895065204 +0000 UTC m=+3933.060550493" watchObservedRunningTime="2026-11-01 15:23:12.897719785 +0000 UTC m=+3933.063205082"
Nov 01 15:23:13.107443 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:13.107378    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:13.113321 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:13.113293    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:13.784882 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:13.784849    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:13.785841 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:13.785817    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:23:16.163834 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:16.163796    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:23:19.352026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.351957    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"]
Nov 01 15:23:19.357142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.357097    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:19.424910 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.424874    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"]
Nov 01 15:23:19.471377 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.471337    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-89e5ef-dockercfg-cwhrs\""
Nov 01 15:23:19.471377 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.471366    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-89e5ef\""
Nov 01 15:23:19.520576 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.520542    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs745\" (UniqueName: \"kubernetes.io/projected/f5605329-e38a-463b-ada1-6f820e1ab519-kube-api-access-rs745\") pod \"payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db\" (UID: \"f5605329-e38a-463b-ada1-6f820e1ab519\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:19.520761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.520695    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5605329-e38a-463b-ada1-6f820e1ab519-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db\" (UID: \"f5605329-e38a-463b-ada1-6f820e1ab519\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:19.621941 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.621861    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5605329-e38a-463b-ada1-6f820e1ab519-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db\" (UID: \"f5605329-e38a-463b-ada1-6f820e1ab519\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:19.622109 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.622023    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs745\" (UniqueName: \"kubernetes.io/projected/f5605329-e38a-463b-ada1-6f820e1ab519-kube-api-access-rs745\") pod \"payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db\" (UID: \"f5605329-e38a-463b-ada1-6f820e1ab519\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:19.622591 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.622574    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5605329-e38a-463b-ada1-6f820e1ab519-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db\" (UID: \"f5605329-e38a-463b-ada1-6f820e1ab519\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:19.679929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.679896    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs745\" (UniqueName: \"kubernetes.io/projected/f5605329-e38a-463b-ada1-6f820e1ab519-kube-api-access-rs745\") pod \"payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db\" (UID: \"f5605329-e38a-463b-ada1-6f820e1ab519\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:19.707660 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:19.707632    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:20.079188 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:23:20.079152    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5605329_e38a_463b_ada1_6f820e1ab519.slice/crio-840cd0241f52eaa358b29270ee5905026e33bc978e983f688f44cc791bfca8fe WatchSource:0}: Error finding container 840cd0241f52eaa358b29270ee5905026e33bc978e983f688f44cc791bfca8fe: Status 404 returned error can't find the container with id 840cd0241f52eaa358b29270ee5905026e33bc978e983f688f44cc791bfca8fe
Nov 01 15:23:20.082523 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:20.082493    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"]
Nov 01 15:23:20.833100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:20.833068    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db" event={"ID":"f5605329-e38a-463b-ada1-6f820e1ab519","Type":"ContainerStarted","Data":"621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071"}
Nov 01 15:23:20.833520 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:20.833131    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:20.833520 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:20.833149    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db" event={"ID":"f5605329-e38a-463b-ada1-6f820e1ab519","Type":"ContainerStarted","Data":"840cd0241f52eaa358b29270ee5905026e33bc978e983f688f44cc791bfca8fe"}
Nov 01 15:23:20.951965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:20.951913    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db" podStartSLOduration=1.951898231 podStartE2EDuration="1.951898231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:23:20.949151324 +0000 UTC m=+3941.114636615" watchObservedRunningTime="2026-11-01 15:23:20.951898231 +0000 UTC m=+3941.117383529"
Nov 01 15:23:26.164080 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:26.164041    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:23:29.959505 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:29.959470    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:23:31.842304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:31.842276    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:23:35.149831 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:23:35.149792    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:23:36.427777 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.427744    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"]
Nov 01 15:23:36.433355 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.433333    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:36.488946 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.488918    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"]
Nov 01 15:23:36.547411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.542698    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-04aa47\""
Nov 01 15:23:36.547411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.542882    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-04aa47-dockercfg-s4j94\""
Nov 01 15:23:36.581411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.552437    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt6zd\" (UniqueName: \"kubernetes.io/projected/f23108a5-4ee7-4586-959b-ef87fb9d8b86-kube-api-access-lt6zd\") pod \"payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8\" (UID: \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:36.581411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.552503    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23108a5-4ee7-4586-959b-ef87fb9d8b86-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8\" (UID: \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:36.728856 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.728759    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt6zd\" (UniqueName: \"kubernetes.io/projected/f23108a5-4ee7-4586-959b-ef87fb9d8b86-kube-api-access-lt6zd\") pod \"payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8\" (UID: \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:36.728856 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.728829    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23108a5-4ee7-4586-959b-ef87fb9d8b86-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8\" (UID: \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:36.732428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.729988    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23108a5-4ee7-4586-959b-ef87fb9d8b86-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8\" (UID: \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:36.788134 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:36.788104    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt6zd\" (UniqueName: \"kubernetes.io/projected/f23108a5-4ee7-4586-959b-ef87fb9d8b86-kube-api-access-lt6zd\") pod \"payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8\" (UID: \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:37.047597 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:37.047519    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:37.417268 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:23:37.417228    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23108a5_4ee7_4586_959b_ef87fb9d8b86.slice/crio-f9ba261962658a38546e8c4984eecc49bdfa5512677f14842d32be054c7d58c5 WatchSource:0}: Error finding container f9ba261962658a38546e8c4984eecc49bdfa5512677f14842d32be054c7d58c5: Status 404 returned error can't find the container with id f9ba261962658a38546e8c4984eecc49bdfa5512677f14842d32be054c7d58c5
Nov 01 15:23:37.421141 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:37.421119    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"]
Nov 01 15:23:37.941631 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:37.941589    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8" event={"ID":"f23108a5-4ee7-4586-959b-ef87fb9d8b86","Type":"ContainerStarted","Data":"e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896"}
Nov 01 15:23:37.942055 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:37.941652    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8" event={"ID":"f23108a5-4ee7-4586-959b-ef87fb9d8b86","Type":"ContainerStarted","Data":"f9ba261962658a38546e8c4984eecc49bdfa5512677f14842d32be054c7d58c5"}
Nov 01 15:23:37.942055 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:37.941694    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:38.056691 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:38.056645    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8" podStartSLOduration=2.056633295 podStartE2EDuration="2.056633295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:23:38.0556174 +0000 UTC m=+3958.221102699" watchObservedRunningTime="2026-11-01 15:23:38.056633295 +0000 UTC m=+3958.222118623"
Nov 01 15:23:39.959764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:39.959728    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:23:48.952527 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:48.952500    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:23:49.959672 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:49.959591    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:23:59.960115 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:23:59.960082    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:24:05.149172 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:24:05.149135    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:24:09.960116 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:09.960075    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:24:11.927806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:11.927765    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:24:19.959779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:19.959732    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:24:22.262450 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:22.262413    5423 generic.go:356] "Generic (PLEG): container finished" podID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerID="6f5112d87654345d95788afede0733d5d34e98c0cbbcde45a3f12ff19b6e7907" exitCode=0
Nov 01 15:24:22.262924 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:22.262468    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" event={"ID":"6427e8b0-629c-444c-b8bc-bc9aa51cc50d","Type":"ContainerDied","Data":"6f5112d87654345d95788afede0733d5d34e98c0cbbcde45a3f12ff19b6e7907"}
Nov 01 15:24:22.262924 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:22.262876    5423 scope.go:122] "RemoveContainer" containerID="6f5112d87654345d95788afede0733d5d34e98c0cbbcde45a3f12ff19b6e7907"
Nov 01 15:24:22.264672 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:22.264641    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:24:22.264760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:22.264728    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:24:22.264798 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:22.264773    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:24:23.279377 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:23.279341    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" event={"ID":"6427e8b0-629c-444c-b8bc-bc9aa51cc50d","Type":"ContainerStarted","Data":"687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6"}
Nov 01 15:24:23.777898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:23.777863    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:24:23.778999 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:23.778960    5423 patch_prober.go:29] interesting pod/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 01 15:24:23.779144 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:23.779029    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 01 15:24:24.777845 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:24.777810    5423 patch_prober.go:29] interesting pod/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 01 15:24:24.778240 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:24.777861    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 01 15:24:25.778307 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:25.778273    5423 patch_prober.go:29] interesting pod/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 01 15:24:25.778725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:25.778325    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 01 15:24:26.778649 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:26.778613    5423 patch_prober.go:29] interesting pod/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 01 15:24:26.779219 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:26.778684    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 01 15:24:27.168699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:27.168659    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:24:27.778460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:27.778425    5423 patch_prober.go:29] interesting pod/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 01 15:24:27.778645 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:27.778486    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 01 15:24:28.783005 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:28.782956    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:24:28.784019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:28.783999    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:24:29.960132 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:29.960099    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:24:34.003729 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:24:34.003693    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-dda3bf67b7e4: secret "agc-cred-dda3bf67b7e4" not found
Nov 01 15:24:34.003729 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:24:34.003727    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:24:34.003729 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:24:34.003737    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: [secret "agc-cred-dda3bf67b7e4" not found, secret "agc-cred-eb52a593c198" not found]
Nov 01 15:24:34.004229 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:24:34.003806    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:26:36.003790959 +0000 UTC m=+4136.169276235 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : [secret "agc-cred-dda3bf67b7e4" not found, secret "agc-cred-eb52a593c198" not found]
Nov 01 15:24:35.150419 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:24:35.150367    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:24:39.960149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:39.960115    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:24:44.318875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:44.318826    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:24:49.958199 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:49.958161    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:24:59.958174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:24:59.958138    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:25:05.150475 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:25:05.150440    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:25:07.523721 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.523685    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"]
Nov 01 15:25:07.528186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.528165    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.567467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.567433    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.567616 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.567499    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-metrics-certs\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.567616 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.567585    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.567770 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.567749    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw4tf\" (UniqueName: \"kubernetes.io/projected/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-kube-api-access-lw4tf\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.585011 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.584985    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"]
Nov 01 15:25:07.632581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.632546    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-04aa47-serving-cert\""
Nov 01 15:25:07.669382 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.669342    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.669843 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:25:07.669723    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-serving-cert: secret "maas-api-e2e-inf-b-04aa47-serving-cert" not found
Nov 01 15:25:07.670322 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:25:07.670301    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls podName:8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e nodeName:}" failed. No retries permitted until 2026-11-01 15:25:08.170275611 +0000 UTC m=+4048.335760894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls") pod "maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc" (UID: "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e") : secret "maas-api-e2e-inf-b-04aa47-serving-cert" not found
Nov 01 15:25:07.670449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.670425    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-metrics-certs\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.670524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.670495    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.670693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.670673    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4tf\" (UniqueName: \"kubernetes.io/projected/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-kube-api-access-lw4tf\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.671481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.671458    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.673613 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.673592    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-metrics-certs\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:07.807470 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:07.807170    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4tf\" (UniqueName: \"kubernetes.io/projected/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-kube-api-access-lw4tf\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:08.193044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:08.192962    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:08.215010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:08.214964    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls\") pod \"maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:08.441783 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:08.441744    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:08.838334 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:25:08.838299    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb99f00_e0ac_41d0_b84d_4d17dd5beb7e.slice/crio-fbdb02bcea31b5a399050ba30331a58d03c2a0199c026fd723f239eea20cffec WatchSource:0}: Error finding container fbdb02bcea31b5a399050ba30331a58d03c2a0199c026fd723f239eea20cffec: Status 404 returned error can't find the container with id fbdb02bcea31b5a399050ba30331a58d03c2a0199c026fd723f239eea20cffec
Nov 01 15:25:08.839860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:08.839839    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:25:08.841685 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:08.841663    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"]
Nov 01 15:25:09.640809 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:09.640779    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc" event={"ID":"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e","Type":"ContainerStarted","Data":"fbdb02bcea31b5a399050ba30331a58d03c2a0199c026fd723f239eea20cffec"}
Nov 01 15:25:09.958551 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:09.958466    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:25:10.655653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:10.655611    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc" event={"ID":"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e","Type":"ContainerStarted","Data":"7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369"}
Nov 01 15:25:10.655919 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:10.655698    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:14.825542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:14.825509    5423 patch_prober.go:29] interesting pod/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:25:14.825932 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:14.825569    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:25:15.699412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:15.699358    5423 generic.go:356] "Generic (PLEG): container finished" podID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerID="75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c" exitCode=0
Nov 01 15:25:15.699627 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:15.699437    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" event={"ID":"9892c3e6-7a37-4d91-b593-a71576e0fd52","Type":"ContainerDied","Data":"75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c"}
Nov 01 15:25:15.699844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:15.699831    5423 scope.go:122] "RemoveContainer" containerID="75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c"
Nov 01 15:25:15.701526 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:15.701504    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:25:15.701594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:15.701569    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:25:15.701631 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:15.701603    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:25:15.913485 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:15.913431    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc" podStartSLOduration=8.028535671 podStartE2EDuration="8.913413651s" totalImagesPullingTime="884.87798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:25:10.833065633 +0000 UTC m=+4050.998550933" watchObservedRunningTime="2026-11-01 15:25:15.913413651 +0000 UTC m=+4056.078898951"
Nov 01 15:25:16.681869 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:16.681833    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:25:16.727070 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:16.727034    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" event={"ID":"9892c3e6-7a37-4d91-b593-a71576e0fd52","Type":"ContainerStarted","Data":"0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0"}
Nov 01 15:25:17.114281 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:17.113824    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:25:17.114281 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:17.113948    5423 patch_prober.go:29] interesting pod/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 01 15:25:17.114281 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:17.113989    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 01 15:25:18.108781 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:18.108742    5423 patch_prober.go:29] interesting pod/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 01 15:25:18.109094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:18.109064    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 01 15:25:19.113787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:19.113746    5423 patch_prober.go:29] interesting pod/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 01 15:25:19.114416 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:19.114364    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 01 15:25:19.958350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:19.958313    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:25:20.110768 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:20.110730    5423 patch_prober.go:29] interesting pod/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 01 15:25:20.111470 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:20.111437    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 01 15:25:21.118768 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:21.118728    5423 patch_prober.go:29] interesting pod/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 01 15:25:21.119398 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:21.119346    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 01 15:25:22.111548 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:22.111518    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:25:22.111952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:22.111934    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:25:22.112548 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:22.112528    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:25:29.958710 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:29.958676    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:25:35.150905 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:25:35.150860    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:25:35.513507 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:35.513427    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:25:39.959367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:39.959328    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:25:49.959694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:49.959659    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:25:51.887623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:51.887582    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"]
Nov 01 15:25:51.919400 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:51.919336    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"]
Nov 01 15:25:56.915229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:56.915180    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx" podUID="9c290225-35e5-412e-9bc4-3f9767008bb4" containerName="payload-processing" containerID="cri-o://b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1" gracePeriod=25
Nov 01 15:25:56.920742 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:56.920707    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v" podUID="75a68c75-a162-42fa-a0f9-486c3dbe91d9" containerName="payload-pre-processing" containerID="cri-o://8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880" gracePeriod=25
Nov 01 15:25:57.270799 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.270773    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:25:57.298585 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.298564    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:25:57.322688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.322659    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a68c75-a162-42fa-a0f9-486c3dbe91d9-kube-api-access-6l4kp\" (UniqueName: \"kubernetes.io/projected/75a68c75-a162-42fa-a0f9-486c3dbe91d9-kube-api-access-6l4kp\") pod \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\" (UID: \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\") "
Nov 01 15:25:57.322798 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.322742    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a68c75-a162-42fa-a0f9-486c3dbe91d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a68c75-a162-42fa-a0f9-486c3dbe91d9-plugins-config-volume\") pod \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\" (UID: \"75a68c75-a162-42fa-a0f9-486c3dbe91d9\") "
Nov 01 15:25:57.323106 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.323076    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a68c75-a162-42fa-a0f9-486c3dbe91d9-plugins-config-volume" pod "75a68c75-a162-42fa-a0f9-486c3dbe91d9" (UID: "75a68c75-a162-42fa-a0f9-486c3dbe91d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:25:57.323361 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.323342    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a68c75-a162-42fa-a0f9-486c3dbe91d9-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:25:57.324775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.324754    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a68c75-a162-42fa-a0f9-486c3dbe91d9-kube-api-access-6l4kp" pod "75a68c75-a162-42fa-a0f9-486c3dbe91d9" (UID: "75a68c75-a162-42fa-a0f9-486c3dbe91d9"). InnerVolumeSpecName "kube-api-access-6l4kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:25:57.424372 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.424285    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c290225-35e5-412e-9bc4-3f9767008bb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290225-35e5-412e-9bc4-3f9767008bb4-plugins-config-volume\") pod \"9c290225-35e5-412e-9bc4-3f9767008bb4\" (UID: \"9c290225-35e5-412e-9bc4-3f9767008bb4\") "
Nov 01 15:25:57.424562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.424466    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c290225-35e5-412e-9bc4-3f9767008bb4-kube-api-access-sj6tq\" (UniqueName: \"kubernetes.io/projected/9c290225-35e5-412e-9bc4-3f9767008bb4-kube-api-access-sj6tq\") pod \"9c290225-35e5-412e-9bc4-3f9767008bb4\" (UID: \"9c290225-35e5-412e-9bc4-3f9767008bb4\") "
Nov 01 15:25:57.424679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.424656    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c290225-35e5-412e-9bc4-3f9767008bb4-plugins-config-volume" pod "9c290225-35e5-412e-9bc4-3f9767008bb4" (UID: "9c290225-35e5-412e-9bc4-3f9767008bb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:25:57.424839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.424825    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c290225-35e5-412e-9bc4-3f9767008bb4-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:25:57.424914 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.424842    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l4kp\" (UniqueName: \"kubernetes.io/projected/75a68c75-a162-42fa-a0f9-486c3dbe91d9-kube-api-access-6l4kp\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:25:57.426583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.426553    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c290225-35e5-412e-9bc4-3f9767008bb4-kube-api-access-sj6tq" pod "9c290225-35e5-412e-9bc4-3f9767008bb4" (UID: "9c290225-35e5-412e-9bc4-3f9767008bb4"). InnerVolumeSpecName "kube-api-access-sj6tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:25:57.525952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:57.525917    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj6tq\" (UniqueName: \"kubernetes.io/projected/9c290225-35e5-412e-9bc4-3f9767008bb4-kube-api-access-sj6tq\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:25:58.021020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.020988    5423 generic.go:356] "Generic (PLEG): container finished" podID="9c290225-35e5-412e-9bc4-3f9767008bb4" containerID="b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1" exitCode=0
Nov 01 15:25:58.021381 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.021031    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"
Nov 01 15:25:58.021381 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.021069    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx" event={"ID":"9c290225-35e5-412e-9bc4-3f9767008bb4","Type":"ContainerDied","Data":"b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1"}
Nov 01 15:25:58.021381 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.021118    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx" event={"ID":"9c290225-35e5-412e-9bc4-3f9767008bb4","Type":"ContainerDied","Data":"97ce2c8acc2ce71fbd8101cca22649ec8504aac94bdcaaad7acf7f03d14b0c75"}
Nov 01 15:25:58.021381 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.021148    5423 scope.go:122] "RemoveContainer" containerID="b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1"
Nov 01 15:25:58.022665 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.022641    5423 generic.go:356] "Generic (PLEG): container finished" podID="75a68c75-a162-42fa-a0f9-486c3dbe91d9" containerID="8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880" exitCode=0
Nov 01 15:25:58.022784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.022700    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"
Nov 01 15:25:58.022860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.022809    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v" event={"ID":"75a68c75-a162-42fa-a0f9-486c3dbe91d9","Type":"ContainerDied","Data":"8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880"}
Nov 01 15:25:58.022860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.022848    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v" event={"ID":"75a68c75-a162-42fa-a0f9-486c3dbe91d9","Type":"ContainerDied","Data":"d4711089a65a4d1f732a1e947fdda7d598c9efe80a3ad20100a8ac5b78b5a976"}
Nov 01 15:25:58.031598 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.031580    5423 scope.go:122] "RemoveContainer" containerID="b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1"
Nov 01 15:25:58.031875 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:25:58.031855    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1\": container with ID starting with b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1 not found: ID does not exist" containerID="b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1"
Nov 01 15:25:58.031945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.031881    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1"} err="failed to get container status \"b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1\": rpc error: code = NotFound desc = could not find container \"b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1\": container with ID starting with b3974ae4f4901af6d2402a78d4c7f9afe83f2f304bcc06d03a2fc462d4b31ce1 not found: ID does not exist"
Nov 01 15:25:58.031945 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.031898    5423 scope.go:122] "RemoveContainer" containerID="8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880"
Nov 01 15:25:58.041002 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.040976    5423 scope.go:122] "RemoveContainer" containerID="8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880"
Nov 01 15:25:58.041278 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:25:58.041259    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880\": container with ID starting with 8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880 not found: ID does not exist" containerID="8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880"
Nov 01 15:25:58.041355 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.041289    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880"} err="failed to get container status \"8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880\": rpc error: code = NotFound desc = could not find container \"8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880\": container with ID starting with 8c80f2141d54b1f5da87250e2538d21574a2235eb641c57c5907e7c3b2aa0880 not found: ID does not exist"
Nov 01 15:25:58.188520 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.188450    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"]
Nov 01 15:25:58.192369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.192345    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5feb76-7c4d7dc7d7-zbq5v"]
Nov 01 15:25:58.304432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.304397    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"]
Nov 01 15:25:58.307177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:58.307146    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5feb76-8668b96475-gl6xx"]
Nov 01 15:25:59.958725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:59.958685    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:25:59.962862 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:59.962836    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a68c75-a162-42fa-a0f9-486c3dbe91d9" path="/var/lib/kubelet/pods/75a68c75-a162-42fa-a0f9-486c3dbe91d9/volumes"
Nov 01 15:25:59.963304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:25:59.963288    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c290225-35e5-412e-9bc4-3f9767008bb4" path="/var/lib/kubelet/pods/9c290225-35e5-412e-9bc4-3f9767008bb4/volumes"
Nov 01 15:26:05.149923 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:26:05.149883    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:26:09.958291 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:09.958248    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:26:11.954853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:11.954816    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:26:21.955284 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:21.955247    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:26:26.081347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:26.081310    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"]
Nov 01 15:26:31.082778 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.082730    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m" podUID="dd83a1fb-1f07-4fb6-9d44-2f2316f4399d" containerName="payload-pre-processing" containerID="cri-o://12f847325d3b8692816ee8de867f348d51ca8780caf8998abfdca6cbf459da62" gracePeriod=25
Nov 01 15:26:31.281683 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.281652    5423 generic.go:356] "Generic (PLEG): container finished" podID="dd83a1fb-1f07-4fb6-9d44-2f2316f4399d" containerID="12f847325d3b8692816ee8de867f348d51ca8780caf8998abfdca6cbf459da62" exitCode=0
Nov 01 15:26:31.281896 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.281691    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m" event={"ID":"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d","Type":"ContainerDied","Data":"12f847325d3b8692816ee8de867f348d51ca8780caf8998abfdca6cbf459da62"}
Nov 01 15:26:31.345378 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.345321    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:26:31.523452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.523376    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-kube-api-access-dlzhv\" (UniqueName: \"kubernetes.io/projected/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-kube-api-access-dlzhv\") pod \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\" (UID: \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\") "
Nov 01 15:26:31.523724 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.523514    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-plugins-config-volume\") pod \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\" (UID: \"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d\") "
Nov 01 15:26:31.523927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.523901    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-plugins-config-volume" pod "dd83a1fb-1f07-4fb6-9d44-2f2316f4399d" (UID: "dd83a1fb-1f07-4fb6-9d44-2f2316f4399d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:26:31.524190 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.524165    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:26:31.525479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.525459    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-kube-api-access-dlzhv" pod "dd83a1fb-1f07-4fb6-9d44-2f2316f4399d" (UID: "dd83a1fb-1f07-4fb6-9d44-2f2316f4399d"). InnerVolumeSpecName "kube-api-access-dlzhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:26:31.625684 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.625615    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlzhv\" (UniqueName: \"kubernetes.io/projected/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d-kube-api-access-dlzhv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:26:31.955927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:31.955841    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:26:32.290104 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:32.290026    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"
Nov 01 15:26:32.290104 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:32.290062    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m" event={"ID":"dd83a1fb-1f07-4fb6-9d44-2f2316f4399d","Type":"ContainerDied","Data":"53476323bf6451e6ffc7ac0f90d2c784749443675a70f9b2aa3a01749cb77887"}
Nov 01 15:26:32.290665 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:32.290121    5423 scope.go:122] "RemoveContainer" containerID="12f847325d3b8692816ee8de867f348d51ca8780caf8998abfdca6cbf459da62"
Nov 01 15:26:32.458273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:32.458237    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"]
Nov 01 15:26:32.509246 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:32.509214    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc8c64-d744dfd95-nfg6m"]
Nov 01 15:26:33.960727 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:33.960693    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd83a1fb-1f07-4fb6-9d44-2f2316f4399d" path="/var/lib/kubelet/pods/dd83a1fb-1f07-4fb6-9d44-2f2316f4399d/volumes"
Nov 01 15:26:35.150011 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:26:35.149968    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:26:36.078479 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:26:36.078440    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-dda3bf67b7e4: secret "agc-cred-dda3bf67b7e4" not found
Nov 01 15:26:36.078479 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:26:36.078472    5423 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb52a593c198: secret "agc-cred-eb52a593c198" not found
Nov 01 15:26:36.078479 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:26:36.078484    5423 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5: [secret "agc-cred-dda3bf67b7e4" not found, secret "agc-cred-eb52a593c198" not found]
Nov 01 15:26:36.078750 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:26:36.078545    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials podName:8c2e9bc3-d6ee-4b1b-a980-0db37b000cab nodeName:}" failed. No retries permitted until 2026-11-01 15:28:38.078529739 +0000 UTC m=+4258.244015028 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials") pod "payload-processing-external-model-5f9f8d5b5-mnrd5" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab") : [secret "agc-cred-dda3bf67b7e4" not found, secret "agc-cred-eb52a593c198" not found]
Nov 01 15:26:41.954873 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:41.954829    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:26:51.955068 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:26:51.954970    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:27:01.955735 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:01.955697    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:27:05.150156 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:27:05.150119    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:27:11.954797 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:11.954761    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:27:18.954615 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:18.954581    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:27:28.954633 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:28.954598    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:27:35.150132 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:27:35.150096    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:27:36.333237 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:36.333204    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"]
Nov 01 15:27:36.334516 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:36.334485    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:27:36.384150 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:36.384118    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"]
Nov 01 15:27:41.335688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.335635    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing" containerID="cri-o://20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d" gracePeriod=25
Nov 01 15:27:41.384735 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.384696    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5" podUID="8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" containerName="payload-processing" containerID="cri-o://d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4" gracePeriod=25
Nov 01 15:27:41.667527 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.667497    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:27:41.683834 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.683809    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-provider-credentials\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-provider-credentials\") pod \"291332d8-abef-4e91-b966-149b19fd1939\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") "
Nov 01 15:27:41.683940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.683904    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-kube-api-access-v4l7g\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-kube-api-access-v4l7g\") pod \"291332d8-abef-4e91-b966-149b19fd1939\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") "
Nov 01 15:27:41.683940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.683931    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-routing-overlay-volume\") pod \"291332d8-abef-4e91-b966-149b19fd1939\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") "
Nov 01 15:27:41.684053 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.684041    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-plugins-config-volume\") pod \"291332d8-abef-4e91-b966-149b19fd1939\" (UID: \"291332d8-abef-4e91-b966-149b19fd1939\") "
Nov 01 15:27:41.684359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.684333    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-routing-overlay-volume" pod "291332d8-abef-4e91-b966-149b19fd1939" (UID: "291332d8-abef-4e91-b966-149b19fd1939"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:27:41.684566 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.684473    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-plugins-config-volume" pod "291332d8-abef-4e91-b966-149b19fd1939" (UID: "291332d8-abef-4e91-b966-149b19fd1939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:27:41.684868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.684646    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:41.684868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.684665    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291332d8-abef-4e91-b966-149b19fd1939-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:41.686208 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.686182    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-provider-credentials" pod "291332d8-abef-4e91-b966-149b19fd1939" (UID: "291332d8-abef-4e91-b966-149b19fd1939"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:27:41.686302 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.686284    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-kube-api-access-v4l7g" pod "291332d8-abef-4e91-b966-149b19fd1939" (UID: "291332d8-abef-4e91-b966-149b19fd1939"). InnerVolumeSpecName "kube-api-access-v4l7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:27:41.694644 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.694621    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:27:41.776566 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.776527    5423 generic.go:356] "Generic (PLEG): container finished" podID="8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" containerID="d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4" exitCode=0
Nov 01 15:27:41.776761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.776571    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"
Nov 01 15:27:41.776761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.776602    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5" event={"ID":"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab","Type":"ContainerDied","Data":"d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4"}
Nov 01 15:27:41.776761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.776664    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5" event={"ID":"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab","Type":"ContainerDied","Data":"b6544511032ee7255cb1f560bea993afb6f5bcbf6653f771fb5df520be45f3a7"}
Nov 01 15:27:41.776761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.776696    5423 scope.go:122] "RemoveContainer" containerID="d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4"
Nov 01 15:27:41.778077 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.778057    5423 generic.go:356] "Generic (PLEG): container finished" podID="291332d8-abef-4e91-b966-149b19fd1939" containerID="20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d" exitCode=0
Nov 01 15:27:41.778198 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.778092    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"
Nov 01 15:27:41.778198 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.778131    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" event={"ID":"291332d8-abef-4e91-b966-149b19fd1939","Type":"ContainerDied","Data":"20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d"}
Nov 01 15:27:41.778198 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.778169    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t" event={"ID":"291332d8-abef-4e91-b966-149b19fd1939","Type":"ContainerDied","Data":"d07494af7312b5c0bba3ab71ab432cde6d93fe709a66bee83a33b4ca6eff4875"}
Nov 01 15:27:41.785232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.785206    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-plugins-config-volume\") pod \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") "
Nov 01 15:27:41.785353 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.785243    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials\") pod \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") "
Nov 01 15:27:41.785353 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.785274    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-routing-overlay-volume\") pod \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") "
Nov 01 15:27:41.785353 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.785336    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-kube-api-access-rqm7s\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-kube-api-access-rqm7s\") pod \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\" (UID: \"8c2e9bc3-d6ee-4b1b-a980-0db37b000cab\") "
Nov 01 15:27:41.785751 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.785726    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-plugins-config-volume" pod "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:27:41.785815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.785767    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-routing-overlay-volume" pod "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:27:41.786035 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.786016    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:41.786035 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.786038    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4l7g\" (UniqueName: \"kubernetes.io/projected/291332d8-abef-4e91-b966-149b19fd1939-kube-api-access-v4l7g\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:41.786218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.786054    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:41.786218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.786068    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:41.787109 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.787080    5423 scope.go:122] "RemoveContainer" containerID="d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4"
Nov 01 15:27:41.787474 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:27:41.787452    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4\": container with ID starting with d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4 not found: ID does not exist" containerID="d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4"
Nov 01 15:27:41.787557 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.787485    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4"} err="failed to get container status \"d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4\": rpc error: code = NotFound desc = could not find container \"d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4\": container with ID starting with d72c506c870c1b047d641986de82bb922a5613c97971b4fbb464ffc585a19ac4 not found: ID does not exist"
Nov 01 15:27:41.787557 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.787507    5423 scope.go:122] "RemoveContainer" containerID="20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d"
Nov 01 15:27:41.788108 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.787974    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-kube-api-access-rqm7s" pod "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab"). InnerVolumeSpecName "kube-api-access-rqm7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:27:41.788108 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.788004    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials" pod "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" (UID: "8c2e9bc3-d6ee-4b1b-a980-0db37b000cab"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:27:41.796280 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.796261    5423 scope.go:122] "RemoveContainer" containerID="20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d"
Nov 01 15:27:41.796556 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:27:41.796538    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d\": container with ID starting with 20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d not found: ID does not exist" containerID="20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d"
Nov 01 15:27:41.796607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.796562    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d"} err="failed to get container status \"20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d\": rpc error: code = NotFound desc = could not find container \"20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d\": container with ID starting with 20e982bf1b34893dbdd40f70937d564793ab613e0c8ddf079b5c255f9e48c98d not found: ID does not exist"
Nov 01 15:27:41.887845 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.887753    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqm7s\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-kube-api-access-rqm7s\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:41.887845 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:41.887783    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:42.105481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:42.105448    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"]
Nov 01 15:27:42.108582 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:42.108556    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b6df859f6-69h8t"]
Nov 01 15:27:42.239671 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:42.239580    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"]
Nov 01 15:27:42.243910 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:42.243875    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f9f8d5b5-mnrd5"]
Nov 01 15:27:43.022454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:43.022429    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:27:43.094824 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:43.094796    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:27:43.960992 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:43.960957    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291332d8-abef-4e91-b966-149b19fd1939" path="/var/lib/kubelet/pods/291332d8-abef-4e91-b966-149b19fd1939/volumes"
Nov 01 15:27:43.961411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:43.961398    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" path="/var/lib/kubelet/pods/8c2e9bc3-d6ee-4b1b-a980-0db37b000cab/volumes"
Nov 01 15:27:51.579715 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:27:51.579676    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:28:00.526809 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:00.526765    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"]
Nov 01 15:28:05.149240 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:28:05.149203    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:28:05.527948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:05.527867    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db" podUID="f5605329-e38a-463b-ada1-6f820e1ab519" containerName="payload-processing" containerID="cri-o://621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071" gracePeriod=25
Nov 01 15:28:05.811769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:05.811742    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:28:05.939798 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:05.939766    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5605329-e38a-463b-ada1-6f820e1ab519-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5605329-e38a-463b-ada1-6f820e1ab519-plugins-config-volume\") pod \"f5605329-e38a-463b-ada1-6f820e1ab519\" (UID: \"f5605329-e38a-463b-ada1-6f820e1ab519\") "
Nov 01 15:28:05.940001 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:05.939852    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5605329-e38a-463b-ada1-6f820e1ab519-kube-api-access-rs745\" (UniqueName: \"kubernetes.io/projected/f5605329-e38a-463b-ada1-6f820e1ab519-kube-api-access-rs745\") pod \"f5605329-e38a-463b-ada1-6f820e1ab519\" (UID: \"f5605329-e38a-463b-ada1-6f820e1ab519\") "
Nov 01 15:28:05.940192 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:05.940164    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5605329-e38a-463b-ada1-6f820e1ab519-plugins-config-volume" pod "f5605329-e38a-463b-ada1-6f820e1ab519" (UID: "f5605329-e38a-463b-ada1-6f820e1ab519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:28:05.940530 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:05.940506    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5605329-e38a-463b-ada1-6f820e1ab519-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:05.942631 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:05.942610    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5605329-e38a-463b-ada1-6f820e1ab519-kube-api-access-rs745" pod "f5605329-e38a-463b-ada1-6f820e1ab519" (UID: "f5605329-e38a-463b-ada1-6f820e1ab519"). InnerVolumeSpecName "kube-api-access-rs745". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:06.041614 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.041550    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs745\" (UniqueName: \"kubernetes.io/projected/f5605329-e38a-463b-ada1-6f820e1ab519-kube-api-access-rs745\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:06.206051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.206016    5423 generic.go:356] "Generic (PLEG): container finished" podID="f5605329-e38a-463b-ada1-6f820e1ab519" containerID="621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071" exitCode=0
Nov 01 15:28:06.206538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.206059    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"
Nov 01 15:28:06.206538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.206096    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db" event={"ID":"f5605329-e38a-463b-ada1-6f820e1ab519","Type":"ContainerDied","Data":"621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071"}
Nov 01 15:28:06.206538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.206151    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db" event={"ID":"f5605329-e38a-463b-ada1-6f820e1ab519","Type":"ContainerDied","Data":"840cd0241f52eaa358b29270ee5905026e33bc978e983f688f44cc791bfca8fe"}
Nov 01 15:28:06.206538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.206174    5423 scope.go:122] "RemoveContainer" containerID="621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071"
Nov 01 15:28:06.216474 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.216457    5423 scope.go:122] "RemoveContainer" containerID="621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071"
Nov 01 15:28:06.216752 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:28:06.216721    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071\": container with ID starting with 621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071 not found: ID does not exist" containerID="621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071"
Nov 01 15:28:06.216816 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.216762    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071"} err="failed to get container status \"621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071\": rpc error: code = NotFound desc = could not find container \"621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071\": container with ID starting with 621f6583b617471b022508d6050a1ed59f327dc0590796701f3f5f07eda08071 not found: ID does not exist"
Nov 01 15:28:06.373619 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.373593    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"]
Nov 01 15:28:06.376070 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:06.376046    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89e5ef-7f69c6dc-5w7db"]
Nov 01 15:28:07.968040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:07.968004    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5605329-e38a-463b-ada1-6f820e1ab519" path="/var/lib/kubelet/pods/f5605329-e38a-463b-ada1-6f820e1ab519/volumes"
Nov 01 15:28:08.391212 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:08.391174    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"]
Nov 01 15:28:08.391476 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:08.391435    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy" containerID="cri-o://8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53" gracePeriod=30
Nov 01 15:28:12.850421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:12.850379    5423 patch_prober.go:29] interesting pod/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:28:12.850824 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:12.850453    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:28:13.658917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.658862    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:28:13.689575 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689545    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-kube-api-access-5v9bk\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689691 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689601    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-podinfo\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689691 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689667    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-socket\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689715    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-data\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689747    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-certs\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689810    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa31553b-417a-48a7-b424-5e4d0135ca4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31553b-417a-48a7-b424-5e4d0135ca4b-istiod-ca-cert\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689983 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689857    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-token\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-token\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689983 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689897    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-envoy\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689983 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689932    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-credential-socket\") pod \"aa31553b-417a-48a7-b424-5e4d0135ca4b\" (UID: \"aa31553b-417a-48a7-b424-5e4d0135ca4b\") "
Nov 01 15:28:13.689983 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689933    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-socket" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:13.690217 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.689980    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-data" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:13.690217 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.690071    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-certs" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:13.690356 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.690329    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-credential-socket" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:13.690445 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.690348    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa31553b-417a-48a7-b424-5e4d0135ca4b-istiod-ca-cert" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:28:13.690580 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.690559    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:13.690632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.690586    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:13.690632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.690602    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31553b-417a-48a7-b424-5e4d0135ca4b-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:13.690632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.690617    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:13.690733 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.690632    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:13.691863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.691840    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-podinfo" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:28:13.692031 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.692008    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-envoy" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:13.692247 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.692230    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-token" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:13.692319 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.692254    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-kube-api-access-5v9bk" pod "aa31553b-417a-48a7-b424-5e4d0135ca4b" (UID: "aa31553b-417a-48a7-b424-5e4d0135ca4b"). InnerVolumeSpecName "kube-api-access-5v9bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:13.791601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.791570    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:13.791601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.791597    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:13.791816 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.791609    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/aa31553b-417a-48a7-b424-5e4d0135ca4b-kube-api-access-5v9bk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:13.791816 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:13.791620    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31553b-417a-48a7-b424-5e4d0135ca4b-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:14.325269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.325234    5423 generic.go:356] "Generic (PLEG): container finished" podID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerID="8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53" exitCode=0
Nov 01 15:28:14.325773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.325297    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"
Nov 01 15:28:14.325773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.325298    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" event={"ID":"aa31553b-417a-48a7-b424-5e4d0135ca4b","Type":"ContainerDied","Data":"8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53"}
Nov 01 15:28:14.325773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.325452    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb" event={"ID":"aa31553b-417a-48a7-b424-5e4d0135ca4b","Type":"ContainerDied","Data":"40afc59161168b1673603b96c4a75024e30b9b047adafda50f7d231599b5fd91"}
Nov 01 15:28:14.325773 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.325487    5423 scope.go:122] "RemoveContainer" containerID="8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53"
Nov 01 15:28:14.335451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.335429    5423 scope.go:122] "RemoveContainer" containerID="82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54"
Nov 01 15:28:14.344836 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.344817    5423 scope.go:122] "RemoveContainer" containerID="8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53"
Nov 01 15:28:14.345084 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:28:14.345066    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53\": container with ID starting with 8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53 not found: ID does not exist" containerID="8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53"
Nov 01 15:28:14.345163 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.345094    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53"} err="failed to get container status \"8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53\": rpc error: code = NotFound desc = could not find container \"8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53\": container with ID starting with 8fe63f5285e70d25c1f80d1a9daa2939f91af7d9f7e4a106d3f5e5c18fc7ec53 not found: ID does not exist"
Nov 01 15:28:14.345163 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.345116    5423 scope.go:122] "RemoveContainer" containerID="82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54"
Nov 01 15:28:14.345365 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:28:14.345345    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54\": container with ID starting with 82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54 not found: ID does not exist" containerID="82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54"
Nov 01 15:28:14.345437 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.345370    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54"} err="failed to get container status \"82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54\": rpc error: code = NotFound desc = could not find container \"82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54\": container with ID starting with 82e8ffe1a2a640ca7c85e0b3f9d321216fdc56fb1f3338a4d00b497aa93b4e54 not found: ID does not exist"
Nov 01 15:28:14.494431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.494375    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"]
Nov 01 15:28:14.545616 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:14.545588    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5a5b-openshift-default-577bb96f5-l2dxb"]
Nov 01 15:28:15.779671 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:15.779635    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"]
Nov 01 15:28:15.966709 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:15.966672    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" path="/var/lib/kubelet/pods/aa31553b-417a-48a7-b424-5e4d0135ca4b/volumes"
Nov 01 15:28:20.780462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:20.780362    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8" podUID="f23108a5-4ee7-4586-959b-ef87fb9d8b86" containerName="payload-pre-processing" containerID="cri-o://e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896" gracePeriod=25
Nov 01 15:28:21.061136 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.061112    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:28:21.188045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.188001    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23108a5-4ee7-4586-959b-ef87fb9d8b86-kube-api-access-lt6zd\" (UniqueName: \"kubernetes.io/projected/f23108a5-4ee7-4586-959b-ef87fb9d8b86-kube-api-access-lt6zd\") pod \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\" (UID: \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\") "
Nov 01 15:28:21.188209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.188174    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23108a5-4ee7-4586-959b-ef87fb9d8b86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23108a5-4ee7-4586-959b-ef87fb9d8b86-plugins-config-volume\") pod \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\" (UID: \"f23108a5-4ee7-4586-959b-ef87fb9d8b86\") "
Nov 01 15:28:21.188595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.188565    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23108a5-4ee7-4586-959b-ef87fb9d8b86-plugins-config-volume" pod "f23108a5-4ee7-4586-959b-ef87fb9d8b86" (UID: "f23108a5-4ee7-4586-959b-ef87fb9d8b86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:28:21.188783 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.188760    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f23108a5-4ee7-4586-959b-ef87fb9d8b86-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:21.190250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.190226    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23108a5-4ee7-4586-959b-ef87fb9d8b86-kube-api-access-lt6zd" pod "f23108a5-4ee7-4586-959b-ef87fb9d8b86" (UID: "f23108a5-4ee7-4586-959b-ef87fb9d8b86"). InnerVolumeSpecName "kube-api-access-lt6zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:21.290262 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.290193    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt6zd\" (UniqueName: \"kubernetes.io/projected/f23108a5-4ee7-4586-959b-ef87fb9d8b86-kube-api-access-lt6zd\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:21.373223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.373193    5423 generic.go:356] "Generic (PLEG): container finished" podID="f23108a5-4ee7-4586-959b-ef87fb9d8b86" containerID="e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896" exitCode=0
Nov 01 15:28:21.373369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.373241    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"
Nov 01 15:28:21.373369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.373285    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8" event={"ID":"f23108a5-4ee7-4586-959b-ef87fb9d8b86","Type":"ContainerDied","Data":"e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896"}
Nov 01 15:28:21.373369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.373341    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8" event={"ID":"f23108a5-4ee7-4586-959b-ef87fb9d8b86","Type":"ContainerDied","Data":"f9ba261962658a38546e8c4984eecc49bdfa5512677f14842d32be054c7d58c5"}
Nov 01 15:28:21.373369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.373363    5423 scope.go:122] "RemoveContainer" containerID="e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896"
Nov 01 15:28:21.383950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.383933    5423 scope.go:122] "RemoveContainer" containerID="e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896"
Nov 01 15:28:21.384210 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:28:21.384194    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896\": container with ID starting with e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896 not found: ID does not exist" containerID="e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896"
Nov 01 15:28:21.384277 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.384217    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896"} err="failed to get container status \"e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896\": rpc error: code = NotFound desc = could not find container \"e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896\": container with ID starting with e98786208ac611a76a73f08848e5ead837a1aed61b69b9cc6bdf2210e4507896 not found: ID does not exist"
Nov 01 15:28:21.540873 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.540789    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"]
Nov 01 15:28:21.543735 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.543709    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-04aa47-6fcbbd9fc5-tq2v8"]
Nov 01 15:28:21.962161 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:21.962132    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23108a5-4ee7-4586-959b-ef87fb9d8b86" path="/var/lib/kubelet/pods/f23108a5-4ee7-4586-959b-ef87fb9d8b86/volumes"
Nov 01 15:28:35.148951 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:28:35.148911    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:28:36.890916 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:36.890881    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"]
Nov 01 15:28:36.891370 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:36.891182    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy" containerID="cri-o://687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6" gracePeriod=30
Nov 01 15:28:36.905879 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:36.905850    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:28:42.160926 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.160874    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:28:42.211053 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211027    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-socket\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211224 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211071    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-credential-socket\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211224 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211125    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-podinfo\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211224 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211152    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-certs\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211436 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211240    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-kube-api-access-x89dv\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-kube-api-access-x89dv\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211436 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211317    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istiod-ca-cert\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211436 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211333    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-socket" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:42.211436 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211381    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-envoy\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211436 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211429    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-data\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211742 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211483    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-token\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-token\") pod \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\" (UID: \"6427e8b0-629c-444c-b8bc-bc9aa51cc50d\") "
Nov 01 15:28:42.211742 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211502    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-credential-socket" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:42.211742 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211547    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-certs" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:42.211742 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211719    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-data" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:42.211962 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211949    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.212023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211969    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.212023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211981    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.212023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.211992    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.212243 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.212043    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istiod-ca-cert" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:28:42.213728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.213701    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-kube-api-access-x89dv" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "kube-api-access-x89dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:42.213826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.213783    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-token" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:42.214755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.214734    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-podinfo" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:28:42.214833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.214785    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-envoy" pod "6427e8b0-629c-444c-b8bc-bc9aa51cc50d" (UID: "6427e8b0-629c-444c-b8bc-bc9aa51cc50d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:28:42.268437 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.268414    5423 scope.go:122] "RemoveContainer" containerID="6f5112d87654345d95788afede0733d5d34e98c0cbbcde45a3f12ff19b6e7907"
Nov 01 15:28:42.313223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.313201    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.313223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.313223    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.313348 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.313233    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.313348 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.313242    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.313348 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.313251    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89dv\" (UniqueName: \"kubernetes.io/projected/6427e8b0-629c-444c-b8bc-bc9aa51cc50d-kube-api-access-x89dv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:42.508125 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.508042    5423 generic.go:356] "Generic (PLEG): container finished" podID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerID="687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6" exitCode=0
Nov 01 15:28:42.508125 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.508096    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"
Nov 01 15:28:42.508125 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.508100    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" event={"ID":"6427e8b0-629c-444c-b8bc-bc9aa51cc50d","Type":"ContainerDied","Data":"687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6"}
Nov 01 15:28:42.508367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.508153    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz" event={"ID":"6427e8b0-629c-444c-b8bc-bc9aa51cc50d","Type":"ContainerDied","Data":"869babec676f31545038f956e8acf89c8e9ad940792f075e05d9e91f481d46a6"}
Nov 01 15:28:42.508367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.508182    5423 scope.go:122] "RemoveContainer" containerID="687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6"
Nov 01 15:28:42.518551 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.518532    5423 scope.go:122] "RemoveContainer" containerID="687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6"
Nov 01 15:28:42.518797 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:28:42.518780    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6\": container with ID starting with 687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6 not found: ID does not exist" containerID="687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6"
Nov 01 15:28:42.518839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.518803    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6"} err="failed to get container status \"687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6\": rpc error: code = NotFound desc = could not find container \"687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6\": container with ID starting with 687779d1e9c1dba31e8b9269b04fdf0b24588ffba90715c96e03010d56c14ca6 not found: ID does not exist"
Nov 01 15:28:42.742760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.742732    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"]
Nov 01 15:28:42.745023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:42.745002    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-775095-openshift-default-5489c5c449-rzcfz"]
Nov 01 15:28:43.960594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:28:43.960561    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" path="/var/lib/kubelet/pods/6427e8b0-629c-444c-b8bc-bc9aa51cc50d/volumes"
Nov 01 15:29:00.908570 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:00.908525    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:29:00.908976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:00.908613    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:29:05.150240 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:29:05.150203    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:29:21.404746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:21.404708    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:29:22.559278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:22.559244    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"]
Nov 01 15:29:22.559739 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:22.559457    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy" containerID="cri-o://0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0" gracePeriod=30
Nov 01 15:29:27.829826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.829768    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:29:27.909487 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.909452    5423 generic.go:356] "Generic (PLEG): container finished" podID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerID="0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0" exitCode=0
Nov 01 15:29:27.909655 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.909572    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"
Nov 01 15:29:27.909732 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.909577    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" event={"ID":"9892c3e6-7a37-4d91-b593-a71576e0fd52","Type":"ContainerDied","Data":"0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0"}
Nov 01 15:29:27.909826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.909739    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g" event={"ID":"9892c3e6-7a37-4d91-b593-a71576e0fd52","Type":"ContainerDied","Data":"20276e3e008c6347e8ecb2363413f7434939d76809c7188b27b4c32b9eae5ed0"}
Nov 01 15:29:27.909826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.909771    5423 scope.go:122] "RemoveContainer" containerID="0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0"
Nov 01 15:29:27.919999 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.919974    5423 scope.go:122] "RemoveContainer" containerID="75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c"
Nov 01 15:29:27.928469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.928443    5423 scope.go:122] "RemoveContainer" containerID="0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0"
Nov 01 15:29:27.928721 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:29:27.928701    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0\": container with ID starting with 0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0 not found: ID does not exist" containerID="0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0"
Nov 01 15:29:27.928777 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.928728    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0"} err="failed to get container status \"0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0\": rpc error: code = NotFound desc = could not find container \"0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0\": container with ID starting with 0fe677c6e5ccb88ffaa8d7b2f5c23829f0c3a7c619dd4047f3bd3dbf04a6a4e0 not found: ID does not exist"
Nov 01 15:29:27.928777 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.928744    5423 scope.go:122] "RemoveContainer" containerID="75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c"
Nov 01 15:29:27.928948 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:29:27.928931    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c\": container with ID starting with 75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c not found: ID does not exist" containerID="75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c"
Nov 01 15:29:27.928995 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.928957    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c"} err="failed to get container status \"75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c\": rpc error: code = NotFound desc = could not find container \"75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c\": container with ID starting with 75d0ac4977dfbede2167296d419d3472958f8ed07de55bc6929b7cf39a82813c not found: ID does not exist"
Nov 01 15:29:27.955884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.955863    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-certs\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.955993 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.955907    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-kube-api-access-cpzg5\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-kube-api-access-cpzg5\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.956061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.955995    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-envoy\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.956061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956043    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-token\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-token\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.956147 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956134    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-credential-socket\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.956186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956141    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-certs" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:29:27.956229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956187    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9892c3e6-7a37-4d91-b593-a71576e0fd52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9892c3e6-7a37-4d91-b593-a71576e0fd52-istiod-ca-cert\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.956229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956217    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-socket\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.956317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956258    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-podinfo\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.956317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956307    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-data\") pod \"9892c3e6-7a37-4d91-b593-a71576e0fd52\" (UID: \"9892c3e6-7a37-4d91-b593-a71576e0fd52\") "
Nov 01 15:29:27.956707 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956681    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-credential-socket" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:29:27.956835 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956693    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9892c3e6-7a37-4d91-b593-a71576e0fd52-istiod-ca-cert" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:29:27.956835 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956695    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-socket" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:29:27.956955 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.956889    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-data" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:29:27.957037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.957014    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:27.957148 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.957039    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9892c3e6-7a37-4d91-b593-a71576e0fd52-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:27.957148 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.957055    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:27.957148 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.957069    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:27.957148 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.957083    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:27.958433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.958409    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-envoy" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:29:27.958550 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.958526    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-podinfo" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:29:27.959231 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.959211    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-kube-api-access-cpzg5" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "kube-api-access-cpzg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:29:27.959886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:27.959863    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-token" pod "9892c3e6-7a37-4d91-b593-a71576e0fd52" (UID: "9892c3e6-7a37-4d91-b593-a71576e0fd52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:29:28.058551 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:28.058525    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:28.058551 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:28.058550    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:28.058708 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:28.058559    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9892c3e6-7a37-4d91-b593-a71576e0fd52-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:28.058708 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:28.058568    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpzg5\" (UniqueName: \"kubernetes.io/projected/9892c3e6-7a37-4d91-b593-a71576e0fd52-kube-api-access-cpzg5\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:28.330771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:28.330740    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"]
Nov 01 15:29:28.335032 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:28.334993    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89e5ef-openshift-default-67dfc48cfc-fvp6g"]
Nov 01 15:29:29.960189 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:29.960145    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" path="/var/lib/kubelet/pods/9892c3e6-7a37-4d91-b593-a71576e0fd52/volumes"
Nov 01 15:29:35.148908 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:29:35.148874    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:29:37.494576 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.494543    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"]
Nov 01 15:29:37.495038 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.494756    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc" podUID="8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" containerName="maas-api" containerID="cri-o://7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369" gracePeriod=30
Nov 01 15:29:37.801995 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.801971    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:29:37.890444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.886789    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-metrics-certs\") pod \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") "
Nov 01 15:29:37.890444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.886893    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-kube-api-access-lw4tf\" (UniqueName: \"kubernetes.io/projected/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-kube-api-access-lw4tf\") pod \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") "
Nov 01 15:29:37.890444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.887002    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-openshift-service-ca\") pod \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") "
Nov 01 15:29:37.890444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.887169    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls\") pod \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\" (UID: \"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e\") "
Nov 01 15:29:37.892060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.890993    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-openshift-service-ca" pod "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" (UID: "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:29:37.900514 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.900463    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls" pod "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" (UID: "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:29:37.900670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.900607    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-kube-api-access-lw4tf" pod "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" (UID: "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e"). InnerVolumeSpecName "kube-api-access-lw4tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:29:37.900670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.900626    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-metrics-certs" pod "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" (UID: "8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:29:37.983977 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.983948    5423 generic.go:356] "Generic (PLEG): container finished" podID="8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" containerID="7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369" exitCode=0
Nov 01 15:29:37.984108 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.984094    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc" event={"ID":"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e","Type":"ContainerDied","Data":"7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369"}
Nov 01 15:29:37.984172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.984129    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc" event={"ID":"8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e","Type":"ContainerDied","Data":"fbdb02bcea31b5a399050ba30331a58d03c2a0199c026fd723f239eea20cffec"}
Nov 01 15:29:37.984172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.984150    5423 scope.go:122] "RemoveContainer" containerID="7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369"
Nov 01 15:29:37.984293 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.984282    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"
Nov 01 15:29:37.988496 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.988471    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:37.988592 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.988498    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw4tf\" (UniqueName: \"kubernetes.io/projected/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-kube-api-access-lw4tf\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:37.988592 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.988512    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:37.988592 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.988526    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:37.996303 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.996282    5423 scope.go:122] "RemoveContainer" containerID="7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369"
Nov 01 15:29:37.996856 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:29:37.996824    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369\": container with ID starting with 7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369 not found: ID does not exist" containerID="7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369"
Nov 01 15:29:37.996937 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:37.996865    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369"} err="failed to get container status \"7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369\": rpc error: code = NotFound desc = could not find container \"7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369\": container with ID starting with 7da77ea03ee15cb58c578c9b6c60ef84d983a8be723ae91917fa125866541369 not found: ID does not exist"
Nov 01 15:29:38.173764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:38.173728    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"]
Nov 01 15:29:38.176689 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:38.176665    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04aa47-74dcbc6f5b-znmnc"]
Nov 01 15:29:39.960710 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:39.960676    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" path="/var/lib/kubelet/pods/8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e/volumes"
Nov 01 15:29:46.630492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:46.628476    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:29:46.630492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:46.628553    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:29:46.942560 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:46.942469    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:29:46.942840 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:29:46.942808    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:30:00.200551 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.200470    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"]
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201071    5423 cpu_manager.go:414] "containerMap: removing container" podUID="dd83a1fb-1f07-4fb6-9d44-2f2316f4399d" containerName="payload-pre-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201086    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd83a1fb-1f07-4fb6-9d44-2f2316f4399d" containerName="payload-pre-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201097    5423 cpu_manager.go:414] "containerMap: removing container" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201103    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201116    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201121    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201136    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9c290225-35e5-412e-9bc4-3f9767008bb4" containerName="payload-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201141    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c290225-35e5-412e-9bc4-3f9767008bb4" containerName="payload-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201148    5423 cpu_manager.go:414] "containerMap: removing container" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201152    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201160    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201165    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201170    5423 cpu_manager.go:414] "containerMap: removing container" podUID="75a68c75-a162-42fa-a0f9-486c3dbe91d9" containerName="payload-pre-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201174    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a68c75-a162-42fa-a0f9-486c3dbe91d9" containerName="payload-pre-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201183    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f23108a5-4ee7-4586-959b-ef87fb9d8b86" containerName="payload-pre-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201188    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23108a5-4ee7-4586-959b-ef87fb9d8b86" containerName="payload-pre-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201204    5423 cpu_manager.go:414] "containerMap: removing container" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201209    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201216    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f5605329-e38a-463b-ada1-6f820e1ab519" containerName="payload-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201220    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5605329-e38a-463b-ada1-6f820e1ab519" containerName="payload-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201232    5423 cpu_manager.go:414] "containerMap: removing container" podUID="8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" containerName="maas-api"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201237    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" containerName="maas-api"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201249    5423 cpu_manager.go:414] "containerMap: removing container" podUID="8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" containerName="payload-processing"
Nov 01 15:30:00.201278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201254    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" containerName="payload-processing"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201333    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201342    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201349    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5605329-e38a-463b-ada1-6f820e1ab519" containerName="payload-processing"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201356    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb99f00-e0ac-41d0-b84d-4d17dd5beb7e" containerName="maas-api"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201363    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2e9bc3-d6ee-4b1b-a980-0db37b000cab" containerName="payload-processing"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201368    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201374    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd83a1fb-1f07-4fb6-9d44-2f2316f4399d" containerName="payload-pre-processing"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201379    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="291332d8-abef-4e91-b966-149b19fd1939" containerName="payload-processing"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201403    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c290225-35e5-412e-9bc4-3f9767008bb4" containerName="payload-processing"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201409    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a68c75-a162-42fa-a0f9-486c3dbe91d9" containerName="payload-pre-processing"
Nov 01 15:30:00.202218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.201416    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23108a5-4ee7-4586-959b-ef87fb9d8b86" containerName="payload-pre-processing"
Nov 01 15:30:00.204816 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.204797    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"]
Nov 01 15:30:00.204933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.204914    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:00.205336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.205321    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy"
Nov 01 15:30:00.205413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.205338    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy"
Nov 01 15:30:00.205413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.205359    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy"
Nov 01 15:30:00.205413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.205364    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy"
Nov 01 15:30:00.205535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.205499    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6427e8b0-629c-444c-b8bc-bc9aa51cc50d" containerName="istio-proxy"
Nov 01 15:30:00.205535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.205510    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa31553b-417a-48a7-b424-5e4d0135ca4b" containerName="istio-proxy"
Nov 01 15:30:00.205535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.205520    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9892c3e6-7a37-4d91-b593-a71576e0fd52" containerName="istio-proxy"
Nov 01 15:30:00.210044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.209980    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:00.218019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.217997    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"]
Nov 01 15:30:00.221951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.221897    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:00.251418 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.251398    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"]
Nov 01 15:30:00.255161 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.255145    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"]
Nov 01 15:30:00.255262 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.255246    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:00.259278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.258981    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:00.332431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.329532    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"]
Nov 01 15:30:00.332431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.331631    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 15:30:00.332431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.331843    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"]
Nov 01 15:30:00.362770 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.362729    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d\" (UID: \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:00.362948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.362778    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de2540-e06f-4a30-8a9f-760ecd705b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879\" (UID: \"e4de2540-e06f-4a30-8a9f-760ecd705b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:00.362948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.362890    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zrc\" (UniqueName: \"kubernetes.io/projected/e4de2540-e06f-4a30-8a9f-760ecd705b55-kube-api-access-m9zrc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879\" (UID: \"e4de2540-e06f-4a30-8a9f-760ecd705b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:00.362948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.362935    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcs6d\" (UniqueName: \"kubernetes.io/projected/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-kube-api-access-zcs6d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d\" (UID: \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:00.363164 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.363046    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb90833-4a70-451b-8267-52dd88ea0205-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892450-4dkfj\" (UID: \"0bb90833-4a70-451b-8267-52dd88ea0205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:00.363164 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.363126    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f20f65-0bca-44ed-b3ab-05e8691623e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf\" (UID: \"83f20f65-0bca-44ed-b3ab-05e8691623e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:00.363164 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.363152    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmt78\" (UniqueName: \"kubernetes.io/projected/0bb90833-4a70-451b-8267-52dd88ea0205-kube-api-access-cmt78\") pod \"maas-api-key-cleanup-29892450-4dkfj\" (UID: \"0bb90833-4a70-451b-8267-52dd88ea0205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:00.363336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.363188    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68fsk\" (UniqueName: \"kubernetes.io/projected/7238518f-23ad-47ec-b759-a7b32827024f-kube-api-access-68fsk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr\" (UID: \"7238518f-23ad-47ec-b759-a7b32827024f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:00.363336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.363225    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7238518f-23ad-47ec-b759-a7b32827024f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr\" (UID: \"7238518f-23ad-47ec-b759-a7b32827024f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:00.363336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.363294    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4zk\" (UniqueName: \"kubernetes.io/projected/83f20f65-0bca-44ed-b3ab-05e8691623e9-kube-api-access-lv4zk\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf\" (UID: \"83f20f65-0bca-44ed-b3ab-05e8691623e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:00.372474 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.372449    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"]
Nov 01 15:30:00.406179 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.405683    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"]
Nov 01 15:30:00.408686 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.408659    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"]
Nov 01 15:30:00.464899 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.464835    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f20f65-0bca-44ed-b3ab-05e8691623e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf\" (UID: \"83f20f65-0bca-44ed-b3ab-05e8691623e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:00.464899 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.464872    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmt78\" (UniqueName: \"kubernetes.io/projected/0bb90833-4a70-451b-8267-52dd88ea0205-kube-api-access-cmt78\") pod \"maas-api-key-cleanup-29892450-4dkfj\" (UID: \"0bb90833-4a70-451b-8267-52dd88ea0205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:00.464899 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.464897    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68fsk\" (UniqueName: \"kubernetes.io/projected/7238518f-23ad-47ec-b759-a7b32827024f-kube-api-access-68fsk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr\" (UID: \"7238518f-23ad-47ec-b759-a7b32827024f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:00.465125 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.464917    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7238518f-23ad-47ec-b759-a7b32827024f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr\" (UID: \"7238518f-23ad-47ec-b759-a7b32827024f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:00.465125 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.465086    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4zk\" (UniqueName: \"kubernetes.io/projected/83f20f65-0bca-44ed-b3ab-05e8691623e9-kube-api-access-lv4zk\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf\" (UID: \"83f20f65-0bca-44ed-b3ab-05e8691623e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:00.465244 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.465220    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d\" (UID: \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:00.465302 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.465257    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de2540-e06f-4a30-8a9f-760ecd705b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879\" (UID: \"e4de2540-e06f-4a30-8a9f-760ecd705b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:00.465359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.465300    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zrc\" (UniqueName: \"kubernetes.io/projected/e4de2540-e06f-4a30-8a9f-760ecd705b55-kube-api-access-m9zrc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879\" (UID: \"e4de2540-e06f-4a30-8a9f-760ecd705b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:00.465438 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.465356    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcs6d\" (UniqueName: \"kubernetes.io/projected/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-kube-api-access-zcs6d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d\" (UID: \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:00.465495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.465460    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb90833-4a70-451b-8267-52dd88ea0205-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892450-4dkfj\" (UID: \"0bb90833-4a70-451b-8267-52dd88ea0205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:00.465814 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.465790    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7238518f-23ad-47ec-b759-a7b32827024f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr\" (UID: \"7238518f-23ad-47ec-b759-a7b32827024f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:00.465917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.465802    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f20f65-0bca-44ed-b3ab-05e8691623e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf\" (UID: \"83f20f65-0bca-44ed-b3ab-05e8691623e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:00.466048 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.466022    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d\" (UID: \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:00.466127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.466028    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de2540-e06f-4a30-8a9f-760ecd705b55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879\" (UID: \"e4de2540-e06f-4a30-8a9f-760ecd705b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:00.466228 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.466209    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb90833-4a70-451b-8267-52dd88ea0205-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892450-4dkfj\" (UID: \"0bb90833-4a70-451b-8267-52dd88ea0205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:00.523928 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.523899    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68fsk\" (UniqueName: \"kubernetes.io/projected/7238518f-23ad-47ec-b759-a7b32827024f-kube-api-access-68fsk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr\" (UID: \"7238518f-23ad-47ec-b759-a7b32827024f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:00.524087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.524035    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4zk\" (UniqueName: \"kubernetes.io/projected/83f20f65-0bca-44ed-b3ab-05e8691623e9-kube-api-access-lv4zk\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf\" (UID: \"83f20f65-0bca-44ed-b3ab-05e8691623e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:00.524149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.524134    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmt78\" (UniqueName: \"kubernetes.io/projected/0bb90833-4a70-451b-8267-52dd88ea0205-kube-api-access-cmt78\") pod \"maas-api-key-cleanup-29892450-4dkfj\" (UID: \"0bb90833-4a70-451b-8267-52dd88ea0205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:00.526911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.526887    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zrc\" (UniqueName: \"kubernetes.io/projected/e4de2540-e06f-4a30-8a9f-760ecd705b55-kube-api-access-m9zrc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879\" (UID: \"e4de2540-e06f-4a30-8a9f-760ecd705b55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:00.526911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.526897    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcs6d\" (UniqueName: \"kubernetes.io/projected/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-kube-api-access-zcs6d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d\" (UID: \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:00.529743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.529723    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:00.537112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.537092    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:00.571129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.571101    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:00.579169 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.579145    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:00.822218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.822190    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:00.911437 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:00.911369    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb90833_4a70_451b_8267_52dd88ea0205.slice/crio-1f506468a19cbffd226d1424b07a37923ad060976442b736065a29cdda25722d WatchSource:0}: Error finding container 1f506468a19cbffd226d1424b07a37923ad060976442b736065a29cdda25722d: Status 404 returned error can't find the container with id 1f506468a19cbffd226d1424b07a37923ad060976442b736065a29cdda25722d
Nov 01 15:30:00.913277 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.913255    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"]
Nov 01 15:30:00.958123 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:00.958092    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5493b1_07ad_40d8_896c_e87e1cf2d09a.slice/crio-c728952e7c09d3922448a02537b5bf7ba9d4aee27b7c1a4190b96f243b89d80b WatchSource:0}: Error finding container c728952e7c09d3922448a02537b5bf7ba9d4aee27b7c1a4190b96f243b89d80b: Status 404 returned error can't find the container with id c728952e7c09d3922448a02537b5bf7ba9d4aee27b7c1a4190b96f243b89d80b
Nov 01 15:30:00.961091 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:00.961068    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"]
Nov 01 15:30:01.005938 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:01.005902    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f20f65_0bca_44ed_b3ab_05e8691623e9.slice/crio-525fad7b7e494be7cd8628128a72ddad03fba88512a507763e854a9518373312 WatchSource:0}: Error finding container 525fad7b7e494be7cd8628128a72ddad03fba88512a507763e854a9518373312: Status 404 returned error can't find the container with id 525fad7b7e494be7cd8628128a72ddad03fba88512a507763e854a9518373312
Nov 01 15:30:01.010041 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:01.010016    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7238518f_23ad_47ec_b759_a7b32827024f.slice/crio-3e6592ecefa36488d6205bc26db65c8679bcec334a9dda1855020787a9e9d33a WatchSource:0}: Error finding container 3e6592ecefa36488d6205bc26db65c8679bcec334a9dda1855020787a9e9d33a: Status 404 returned error can't find the container with id 3e6592ecefa36488d6205bc26db65c8679bcec334a9dda1855020787a9e9d33a
Nov 01 15:30:01.010602 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:01.010582    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"]
Nov 01 15:30:01.013147 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:01.013126    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"]
Nov 01 15:30:01.142684 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:01.142652    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" event={"ID":"7238518f-23ad-47ec-b759-a7b32827024f","Type":"ContainerStarted","Data":"3e6592ecefa36488d6205bc26db65c8679bcec334a9dda1855020787a9e9d33a"}
Nov 01 15:30:01.144020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:01.143990    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" event={"ID":"3d5493b1-07ad-40d8-896c-e87e1cf2d09a","Type":"ContainerStarted","Data":"c728952e7c09d3922448a02537b5bf7ba9d4aee27b7c1a4190b96f243b89d80b"}
Nov 01 15:30:01.145217 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:01.145194    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj" event={"ID":"0bb90833-4a70-451b-8267-52dd88ea0205","Type":"ContainerStarted","Data":"1f506468a19cbffd226d1424b07a37923ad060976442b736065a29cdda25722d"}
Nov 01 15:30:01.146416 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:01.146377    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" event={"ID":"83f20f65-0bca-44ed-b3ab-05e8691623e9","Type":"ContainerStarted","Data":"525fad7b7e494be7cd8628128a72ddad03fba88512a507763e854a9518373312"}
Nov 01 15:30:01.194226 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:01.194207    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"]
Nov 01 15:30:01.197658 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:01.197636    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4de2540_e06f_4a30_8a9f_760ecd705b55.slice/crio-3a596f31d4139faa7705847112ea71d48eca89f0e9ca659d29afa04aa13091c2 WatchSource:0}: Error finding container 3a596f31d4139faa7705847112ea71d48eca89f0e9ca659d29afa04aa13091c2: Status 404 returned error can't find the container with id 3a596f31d4139faa7705847112ea71d48eca89f0e9ca659d29afa04aa13091c2
Nov 01 15:30:02.155511 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.155024    5423 generic.go:356] "Generic (PLEG): container finished" podID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerID="70b285e6a1579abf5673fc4987b3268dd2e8e493f1a9b16edea8e537cd7ae3ae" exitCode=60
Nov 01 15:30:02.155511 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.155138    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" event={"ID":"83f20f65-0bca-44ed-b3ab-05e8691623e9","Type":"ContainerDied","Data":"70b285e6a1579abf5673fc4987b3268dd2e8e493f1a9b16edea8e537cd7ae3ae"}
Nov 01 15:30:02.156004 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.155582    5423 scope.go:122] "RemoveContainer" containerID="70b285e6a1579abf5673fc4987b3268dd2e8e493f1a9b16edea8e537cd7ae3ae"
Nov 01 15:30:02.159250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.159224    5423 generic.go:356] "Generic (PLEG): container finished" podID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerID="923d59028785347df7fef2e60199bf672869fe02ab8ff4c50e0d0fd1bdc28bd7" exitCode=60
Nov 01 15:30:02.159509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.159260    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" event={"ID":"e4de2540-e06f-4a30-8a9f-760ecd705b55","Type":"ContainerDied","Data":"923d59028785347df7fef2e60199bf672869fe02ab8ff4c50e0d0fd1bdc28bd7"}
Nov 01 15:30:02.159677 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.159649    5423 scope.go:122] "RemoveContainer" containerID="923d59028785347df7fef2e60199bf672869fe02ab8ff4c50e0d0fd1bdc28bd7"
Nov 01 15:30:02.159677 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.159648    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" event={"ID":"e4de2540-e06f-4a30-8a9f-760ecd705b55","Type":"ContainerStarted","Data":"3a596f31d4139faa7705847112ea71d48eca89f0e9ca659d29afa04aa13091c2"}
Nov 01 15:30:02.161552 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.161535    5423 generic.go:356] "Generic (PLEG): container finished" podID="7238518f-23ad-47ec-b759-a7b32827024f" containerID="287b0cff9ed82407bc61067766a42c3cdbdc5567ab867ea7b0a8e51e86d6e190" exitCode=60
Nov 01 15:30:02.161651 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.161624    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" event={"ID":"7238518f-23ad-47ec-b759-a7b32827024f","Type":"ContainerDied","Data":"287b0cff9ed82407bc61067766a42c3cdbdc5567ab867ea7b0a8e51e86d6e190"}
Nov 01 15:30:02.161864 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.161849    5423 scope.go:122] "RemoveContainer" containerID="287b0cff9ed82407bc61067766a42c3cdbdc5567ab867ea7b0a8e51e86d6e190"
Nov 01 15:30:02.163433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.163415    5423 generic.go:356] "Generic (PLEG): container finished" podID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerID="7dc9d2f55511a1e238dd2a0155ab705d7653a09c2e455384664f932212c37f7f" exitCode=60
Nov 01 15:30:02.163512 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.163493    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" event={"ID":"3d5493b1-07ad-40d8-896c-e87e1cf2d09a","Type":"ContainerDied","Data":"7dc9d2f55511a1e238dd2a0155ab705d7653a09c2e455384664f932212c37f7f"}
Nov 01 15:30:02.163709 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.163697    5423 scope.go:122] "RemoveContainer" containerID="7dc9d2f55511a1e238dd2a0155ab705d7653a09c2e455384664f932212c37f7f"
Nov 01 15:30:02.165111 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.165092    5423 generic.go:356] "Generic (PLEG): container finished" podID="0bb90833-4a70-451b-8267-52dd88ea0205" containerID="6561c4b1d1e1ff5f86f665ed0a1daf583818f60a8dadee86189f0b418b7a53af" exitCode=60
Nov 01 15:30:02.165212 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.165127    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj" event={"ID":"0bb90833-4a70-451b-8267-52dd88ea0205","Type":"ContainerDied","Data":"6561c4b1d1e1ff5f86f665ed0a1daf583818f60a8dadee86189f0b418b7a53af"}
Nov 01 15:30:02.165436 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:02.165417    5423 scope.go:122] "RemoveContainer" containerID="6561c4b1d1e1ff5f86f665ed0a1daf583818f60a8dadee86189f0b418b7a53af"
Nov 01 15:30:03.174645 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.174602    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" event={"ID":"7238518f-23ad-47ec-b759-a7b32827024f","Type":"ContainerStarted","Data":"15757af6d64efa1e71dab3b567ec96b155d15830e217c2cd1ce4d4a06cb67dc9"}
Nov 01 15:30:03.177350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.177319    5423 generic.go:356] "Generic (PLEG): container finished" podID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerID="f4023c4a83fe6a9d10257d8c9eb44a87f0745fed94bd7cac330197e4a0524065" exitCode=60
Nov 01 15:30:03.177471 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.177461    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" event={"ID":"3d5493b1-07ad-40d8-896c-e87e1cf2d09a","Type":"ContainerDied","Data":"f4023c4a83fe6a9d10257d8c9eb44a87f0745fed94bd7cac330197e4a0524065"}
Nov 01 15:30:03.177534 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.177503    5423 scope.go:122] "RemoveContainer" containerID="7dc9d2f55511a1e238dd2a0155ab705d7653a09c2e455384664f932212c37f7f"
Nov 01 15:30:03.177725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.177707    5423 scope.go:122] "RemoveContainer" containerID="f4023c4a83fe6a9d10257d8c9eb44a87f0745fed94bd7cac330197e4a0524065"
Nov 01 15:30:03.178143 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:03.178006    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d_odh-ai-gateway-infra(3d5493b1-07ad-40d8-896c-e87e1cf2d09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a"
Nov 01 15:30:03.179680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.179658    5423 generic.go:356] "Generic (PLEG): container finished" podID="0bb90833-4a70-451b-8267-52dd88ea0205" containerID="a6760a6eb0c4d2654cfacc54869550934c57749c0a7910f9f9758d2885523a46" exitCode=60
Nov 01 15:30:03.179771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.179739    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj" event={"ID":"0bb90833-4a70-451b-8267-52dd88ea0205","Type":"ContainerDied","Data":"a6760a6eb0c4d2654cfacc54869550934c57749c0a7910f9f9758d2885523a46"}
Nov 01 15:30:03.179999 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.179982    5423 scope.go:122] "RemoveContainer" containerID="a6760a6eb0c4d2654cfacc54869550934c57749c0a7910f9f9758d2885523a46"
Nov 01 15:30:03.180288 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:03.180256    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892450-4dkfj_odh-ai-gateway-infra(0bb90833-4a70-451b-8267-52dd88ea0205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj" podUID="0bb90833-4a70-451b-8267-52dd88ea0205"
Nov 01 15:30:03.181727 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.181706    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" event={"ID":"83f20f65-0bca-44ed-b3ab-05e8691623e9","Type":"ContainerStarted","Data":"f7e99d33c25b20cb7fece03377347f79e31557c2a416ff14240b82f3ca256723"}
Nov 01 15:30:03.183692 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.183668    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" event={"ID":"e4de2540-e06f-4a30-8a9f-760ecd705b55","Type":"ContainerStarted","Data":"110db307030db1fba08c313a8d6ac9e3031f650679a69100222dfc087aeb7b74"}
Nov 01 15:30:03.290989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.290939    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" podStartSLOduration=3.290925524 podStartE2EDuration="3.290925524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:03.28918924 +0000 UTC m=+4343.454674536" watchObservedRunningTime="2026-11-01 15:30:03.290925524 +0000 UTC m=+4343.456410814"
Nov 01 15:30:03.329704 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.329681    5423 scope.go:122] "RemoveContainer" containerID="6561c4b1d1e1ff5f86f665ed0a1daf583818f60a8dadee86189f0b418b7a53af"
Nov 01 15:30:03.520803 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.520758    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" podStartSLOduration=3.520744853 podStartE2EDuration="3.520744853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:03.52044139 +0000 UTC m=+4343.685926686" watchObservedRunningTime="2026-11-01 15:30:03.520744853 +0000 UTC m=+4343.686230151"
Nov 01 15:30:03.638428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:03.638333    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" podStartSLOduration=3.638320993 podStartE2EDuration="3.638320993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:03.636727824 +0000 UTC m=+4343.802213115" watchObservedRunningTime="2026-11-01 15:30:03.638320993 +0000 UTC m=+4343.803806290"
Nov 01 15:30:04.192221 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.192193    5423 scope.go:122] "RemoveContainer" containerID="a6760a6eb0c4d2654cfacc54869550934c57749c0a7910f9f9758d2885523a46"
Nov 01 15:30:04.192708 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:04.192542    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892450-4dkfj_odh-ai-gateway-infra(0bb90833-4a70-451b-8267-52dd88ea0205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj" podUID="0bb90833-4a70-451b-8267-52dd88ea0205"
Nov 01 15:30:04.193933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.193911    5423 generic.go:356] "Generic (PLEG): container finished" podID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerID="f7e99d33c25b20cb7fece03377347f79e31557c2a416ff14240b82f3ca256723" exitCode=60
Nov 01 15:30:04.194021 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.193975    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" event={"ID":"83f20f65-0bca-44ed-b3ab-05e8691623e9","Type":"ContainerDied","Data":"f7e99d33c25b20cb7fece03377347f79e31557c2a416ff14240b82f3ca256723"}
Nov 01 15:30:04.194093 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.194023    5423 scope.go:122] "RemoveContainer" containerID="70b285e6a1579abf5673fc4987b3268dd2e8e493f1a9b16edea8e537cd7ae3ae"
Nov 01 15:30:04.194270 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.194252    5423 scope.go:122] "RemoveContainer" containerID="f7e99d33c25b20cb7fece03377347f79e31557c2a416ff14240b82f3ca256723"
Nov 01 15:30:04.194565 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:04.194539    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf_odh-ai-gateway-infra(83f20f65-0bca-44ed-b3ab-05e8691623e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9"
Nov 01 15:30:04.196297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.196261    5423 generic.go:356] "Generic (PLEG): container finished" podID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerID="110db307030db1fba08c313a8d6ac9e3031f650679a69100222dfc087aeb7b74" exitCode=60
Nov 01 15:30:04.196429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.196335    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" event={"ID":"e4de2540-e06f-4a30-8a9f-760ecd705b55","Type":"ContainerDied","Data":"110db307030db1fba08c313a8d6ac9e3031f650679a69100222dfc087aeb7b74"}
Nov 01 15:30:04.196643 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.196626    5423 scope.go:122] "RemoveContainer" containerID="110db307030db1fba08c313a8d6ac9e3031f650679a69100222dfc087aeb7b74"
Nov 01 15:30:04.196917 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:04.196890    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879_odh-ai-gateway-infra(e4de2540-e06f-4a30-8a9f-760ecd705b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55"
Nov 01 15:30:04.198439 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.198421    5423 generic.go:356] "Generic (PLEG): container finished" podID="7238518f-23ad-47ec-b759-a7b32827024f" containerID="15757af6d64efa1e71dab3b567ec96b155d15830e217c2cd1ce4d4a06cb67dc9" exitCode=60
Nov 01 15:30:04.198543 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.198503    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" event={"ID":"7238518f-23ad-47ec-b759-a7b32827024f","Type":"ContainerDied","Data":"15757af6d64efa1e71dab3b567ec96b155d15830e217c2cd1ce4d4a06cb67dc9"}
Nov 01 15:30:04.198719 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.198704    5423 scope.go:122] "RemoveContainer" containerID="15757af6d64efa1e71dab3b567ec96b155d15830e217c2cd1ce4d4a06cb67dc9"
Nov 01 15:30:04.198939 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:04.198914    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr_odh-ai-gateway-infra(7238518f-23ad-47ec-b759-a7b32827024f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" podUID="7238518f-23ad-47ec-b759-a7b32827024f"
Nov 01 15:30:04.200549 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.200532    5423 scope.go:122] "RemoveContainer" containerID="f4023c4a83fe6a9d10257d8c9eb44a87f0745fed94bd7cac330197e4a0524065"
Nov 01 15:30:04.200745 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:04.200729    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d_odh-ai-gateway-infra(3d5493b1-07ad-40d8-896c-e87e1cf2d09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a"
Nov 01 15:30:04.207923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.207905    5423 scope.go:122] "RemoveContainer" containerID="923d59028785347df7fef2e60199bf672869fe02ab8ff4c50e0d0fd1bdc28bd7"
Nov 01 15:30:04.219261 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:04.219243    5423 scope.go:122] "RemoveContainer" containerID="287b0cff9ed82407bc61067766a42c3cdbdc5567ab867ea7b0a8e51e86d6e190"
Nov 01 15:30:05.148533 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:05.148500    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:30:05.208046 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:05.208006    5423 scope.go:122] "RemoveContainer" containerID="f7e99d33c25b20cb7fece03377347f79e31557c2a416ff14240b82f3ca256723"
Nov 01 15:30:05.208641 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:05.208376    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf_odh-ai-gateway-infra(83f20f65-0bca-44ed-b3ab-05e8691623e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9"
Nov 01 15:30:05.210962 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:05.210941    5423 scope.go:122] "RemoveContainer" containerID="110db307030db1fba08c313a8d6ac9e3031f650679a69100222dfc087aeb7b74"
Nov 01 15:30:05.211254 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:05.211234    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879_odh-ai-gateway-infra(e4de2540-e06f-4a30-8a9f-760ecd705b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55"
Nov 01 15:30:05.212687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:05.212670    5423 scope.go:122] "RemoveContainer" containerID="15757af6d64efa1e71dab3b567ec96b155d15830e217c2cd1ce4d4a06cb67dc9"
Nov 01 15:30:05.212902 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:05.212883    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr_odh-ai-gateway-infra(7238518f-23ad-47ec-b759-a7b32827024f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" podUID="7238518f-23ad-47ec-b759-a7b32827024f"
Nov 01 15:30:13.954128 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:13.954096    5423 scope.go:122] "RemoveContainer" containerID="15757af6d64efa1e71dab3b567ec96b155d15830e217c2cd1ce4d4a06cb67dc9"
Nov 01 15:30:13.954634 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:13.954314    5423 scope.go:122] "RemoveContainer" containerID="a6760a6eb0c4d2654cfacc54869550934c57749c0a7910f9f9758d2885523a46"
Nov 01 15:30:14.232428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:14.232401    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:30:15.276427 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.276364    5423 generic.go:356] "Generic (PLEG): container finished" podID="0bb90833-4a70-451b-8267-52dd88ea0205" containerID="09d476e4826ceda2aeb1d5fb09042c9dad930753897da737b72cebcad689c24a" exitCode=60
Nov 01 15:30:15.279042 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.276429    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj" event={"ID":"0bb90833-4a70-451b-8267-52dd88ea0205","Type":"ContainerDied","Data":"09d476e4826ceda2aeb1d5fb09042c9dad930753897da737b72cebcad689c24a"}
Nov 01 15:30:15.279042 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.276505    5423 scope.go:122] "RemoveContainer" containerID="a6760a6eb0c4d2654cfacc54869550934c57749c0a7910f9f9758d2885523a46"
Nov 01 15:30:15.279042 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.276998    5423 scope.go:122] "RemoveContainer" containerID="09d476e4826ceda2aeb1d5fb09042c9dad930753897da737b72cebcad689c24a"
Nov 01 15:30:15.279042 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:15.277349    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892450-4dkfj_odh-ai-gateway-infra(0bb90833-4a70-451b-8267-52dd88ea0205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj" podUID="0bb90833-4a70-451b-8267-52dd88ea0205"
Nov 01 15:30:15.280002 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.279966    5423 generic.go:356] "Generic (PLEG): container finished" podID="7238518f-23ad-47ec-b759-a7b32827024f" containerID="17766a1009cb0953a871226dabcb98cd92cd17ed771e8e6f40a4e99ed9f37b3d" exitCode=60
Nov 01 15:30:15.280112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.280069    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" event={"ID":"7238518f-23ad-47ec-b759-a7b32827024f","Type":"ContainerDied","Data":"17766a1009cb0953a871226dabcb98cd92cd17ed771e8e6f40a4e99ed9f37b3d"}
Nov 01 15:30:15.280545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.280492    5423 scope.go:122] "RemoveContainer" containerID="17766a1009cb0953a871226dabcb98cd92cd17ed771e8e6f40a4e99ed9f37b3d"
Nov 01 15:30:15.280848 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:15.280814    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr_odh-ai-gateway-infra(7238518f-23ad-47ec-b759-a7b32827024f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" podUID="7238518f-23ad-47ec-b759-a7b32827024f"
Nov 01 15:30:15.293669 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.293599    5423 scope.go:122] "RemoveContainer" containerID="15757af6d64efa1e71dab3b567ec96b155d15830e217c2cd1ce4d4a06cb67dc9"
Nov 01 15:30:15.969408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.968844    5423 scope.go:122] "RemoveContainer" containerID="f7e99d33c25b20cb7fece03377347f79e31557c2a416ff14240b82f3ca256723"
Nov 01 15:30:15.970409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:15.969748    5423 scope.go:122] "RemoveContainer" containerID="110db307030db1fba08c313a8d6ac9e3031f650679a69100222dfc087aeb7b74"
Nov 01 15:30:16.460499 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:16.460464    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"]
Nov 01 15:30:16.588924 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:16.587530    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"]
Nov 01 15:30:16.981770 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:16.981287    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:30:16.981770 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:16.981365    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:30:17.424378 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.424082    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:17.427948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.427858    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:17.446195 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.444765    5423 generic.go:356] "Generic (PLEG): container finished" podID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerID="ac3c656ddc8747f355132b9de2d7566ab9baee3dc43696fd5190443a049f4856" exitCode=60
Nov 01 15:30:17.446195 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.444899    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" event={"ID":"e4de2540-e06f-4a30-8a9f-760ecd705b55","Type":"ContainerDied","Data":"ac3c656ddc8747f355132b9de2d7566ab9baee3dc43696fd5190443a049f4856"}
Nov 01 15:30:17.446195 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.444954    5423 scope.go:122] "RemoveContainer" containerID="110db307030db1fba08c313a8d6ac9e3031f650679a69100222dfc087aeb7b74"
Nov 01 15:30:17.446195 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.445455    5423 scope.go:122] "RemoveContainer" containerID="ac3c656ddc8747f355132b9de2d7566ab9baee3dc43696fd5190443a049f4856"
Nov 01 15:30:17.446195 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:17.445802    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879_odh-ai-gateway-infra(e4de2540-e06f-4a30-8a9f-760ecd705b55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55"
Nov 01 15:30:17.450349 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.450324    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr" event={"ID":"7238518f-23ad-47ec-b759-a7b32827024f","Type":"ContainerDied","Data":"3e6592ecefa36488d6205bc26db65c8679bcec334a9dda1855020787a9e9d33a"}
Nov 01 15:30:17.450648 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.450629    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"
Nov 01 15:30:17.456447 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.454063    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj" event={"ID":"0bb90833-4a70-451b-8267-52dd88ea0205","Type":"ContainerDied","Data":"1f506468a19cbffd226d1424b07a37923ad060976442b736065a29cdda25722d"}
Nov 01 15:30:17.456447 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.454153    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"
Nov 01 15:30:17.461405 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.460638    5423 generic.go:356] "Generic (PLEG): container finished" podID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerID="9f165dac3a4f612cbace59055c8dcc6d2085f058ded315f289183fc5b56012fe" exitCode=60
Nov 01 15:30:17.461405 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.460672    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" event={"ID":"83f20f65-0bca-44ed-b3ab-05e8691623e9","Type":"ContainerDied","Data":"9f165dac3a4f612cbace59055c8dcc6d2085f058ded315f289183fc5b56012fe"}
Nov 01 15:30:17.461405 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.461066    5423 scope.go:122] "RemoveContainer" containerID="9f165dac3a4f612cbace59055c8dcc6d2085f058ded315f289183fc5b56012fe"
Nov 01 15:30:17.462105 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:17.461381    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf_odh-ai-gateway-infra(83f20f65-0bca-44ed-b3ab-05e8691623e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9"
Nov 01 15:30:17.504611 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.504582    5423 scope.go:122] "RemoveContainer" containerID="17766a1009cb0953a871226dabcb98cd92cd17ed771e8e6f40a4e99ed9f37b3d"
Nov 01 15:30:17.526919 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.526506    5423 scope.go:122] "RemoveContainer" containerID="09d476e4826ceda2aeb1d5fb09042c9dad930753897da737b72cebcad689c24a"
Nov 01 15:30:17.545099 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.545058    5423 scope.go:122] "RemoveContainer" containerID="f7e99d33c25b20cb7fece03377347f79e31557c2a416ff14240b82f3ca256723"
Nov 01 15:30:17.572408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.571974    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb90833-4a70-451b-8267-52dd88ea0205-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb90833-4a70-451b-8267-52dd88ea0205-maas-api-service-ca\") pod \"0bb90833-4a70-451b-8267-52dd88ea0205\" (UID: \"0bb90833-4a70-451b-8267-52dd88ea0205\") "
Nov 01 15:30:17.572408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.572026    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7238518f-23ad-47ec-b759-a7b32827024f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7238518f-23ad-47ec-b759-a7b32827024f-maas-api-service-ca\") pod \"7238518f-23ad-47ec-b759-a7b32827024f\" (UID: \"7238518f-23ad-47ec-b759-a7b32827024f\") "
Nov 01 15:30:17.572408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.572088    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7238518f-23ad-47ec-b759-a7b32827024f-kube-api-access-68fsk\" (UniqueName: \"kubernetes.io/projected/7238518f-23ad-47ec-b759-a7b32827024f-kube-api-access-68fsk\") pod \"7238518f-23ad-47ec-b759-a7b32827024f\" (UID: \"7238518f-23ad-47ec-b759-a7b32827024f\") "
Nov 01 15:30:17.572408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.572125    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb90833-4a70-451b-8267-52dd88ea0205-kube-api-access-cmt78\" (UniqueName: \"kubernetes.io/projected/0bb90833-4a70-451b-8267-52dd88ea0205-kube-api-access-cmt78\") pod \"0bb90833-4a70-451b-8267-52dd88ea0205\" (UID: \"0bb90833-4a70-451b-8267-52dd88ea0205\") "
Nov 01 15:30:17.574488 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.573365    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb90833-4a70-451b-8267-52dd88ea0205-maas-api-service-ca" pod "0bb90833-4a70-451b-8267-52dd88ea0205" (UID: "0bb90833-4a70-451b-8267-52dd88ea0205"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:30:17.574488 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.573833    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7238518f-23ad-47ec-b759-a7b32827024f-maas-api-service-ca" pod "7238518f-23ad-47ec-b759-a7b32827024f" (UID: "7238518f-23ad-47ec-b759-a7b32827024f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:30:17.579269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.579222    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7238518f-23ad-47ec-b759-a7b32827024f-kube-api-access-68fsk" pod "7238518f-23ad-47ec-b759-a7b32827024f" (UID: "7238518f-23ad-47ec-b759-a7b32827024f"). InnerVolumeSpecName "kube-api-access-68fsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:30:17.589853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.586532    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb90833-4a70-451b-8267-52dd88ea0205-kube-api-access-cmt78" pod "0bb90833-4a70-451b-8267-52dd88ea0205" (UID: "0bb90833-4a70-451b-8267-52dd88ea0205"). InnerVolumeSpecName "kube-api-access-cmt78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:30:17.674054 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.673956    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb90833-4a70-451b-8267-52dd88ea0205-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:17.674054 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.673986    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7238518f-23ad-47ec-b759-a7b32827024f-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:17.674054 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.674000    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68fsk\" (UniqueName: \"kubernetes.io/projected/7238518f-23ad-47ec-b759-a7b32827024f-kube-api-access-68fsk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:17.674054 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.674011    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmt78\" (UniqueName: \"kubernetes.io/projected/0bb90833-4a70-451b-8267-52dd88ea0205-kube-api-access-cmt78\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:17.953757 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:17.953644    5423 scope.go:122] "RemoveContainer" containerID="f4023c4a83fe6a9d10257d8c9eb44a87f0745fed94bd7cac330197e4a0524065"
Nov 01 15:30:18.072085 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.072048    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"]
Nov 01 15:30:18.074701 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.074672    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892450-4dkfj"]
Nov 01 15:30:18.187806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.187769    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"]
Nov 01 15:30:18.192573 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.192546    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892450-p99jr"]
Nov 01 15:30:18.468594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.468511    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" event={"ID":"3d5493b1-07ad-40d8-896c-e87e1cf2d09a","Type":"ContainerStarted","Data":"5884458ef084e665e775a492116b734dbcade9a45fa51777df218dfb44c394c1"}
Nov 01 15:30:18.591138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.591068    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" podStartSLOduration=18.591048692 podStartE2EDuration="18.591048692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:18.587861116 +0000 UTC m=+4358.753346408" watchObservedRunningTime="2026-11-01 15:30:18.591048692 +0000 UTC m=+4358.756533991"
Nov 01 15:30:18.735737 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.735666    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"]
Nov 01 15:30:18.913585 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.913560    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:18.967256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.967213    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"]
Nov 01 15:30:18.988670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.988576    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4de2540-e06f-4a30-8a9f-760ecd705b55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de2540-e06f-4a30-8a9f-760ecd705b55-maas-api-service-ca\") pod \"e4de2540-e06f-4a30-8a9f-760ecd705b55\" (UID: \"e4de2540-e06f-4a30-8a9f-760ecd705b55\") "
Nov 01 15:30:18.988874 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.988674    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4de2540-e06f-4a30-8a9f-760ecd705b55-kube-api-access-m9zrc\" (UniqueName: \"kubernetes.io/projected/e4de2540-e06f-4a30-8a9f-760ecd705b55-kube-api-access-m9zrc\") pod \"e4de2540-e06f-4a30-8a9f-760ecd705b55\" (UID: \"e4de2540-e06f-4a30-8a9f-760ecd705b55\") "
Nov 01 15:30:18.989483 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.989438    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4de2540-e06f-4a30-8a9f-760ecd705b55-maas-api-service-ca" pod "e4de2540-e06f-4a30-8a9f-760ecd705b55" (UID: "e4de2540-e06f-4a30-8a9f-760ecd705b55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:30:18.991906 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:18.991878    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4de2540-e06f-4a30-8a9f-760ecd705b55-kube-api-access-m9zrc" pod "e4de2540-e06f-4a30-8a9f-760ecd705b55" (UID: "e4de2540-e06f-4a30-8a9f-760ecd705b55"). InnerVolumeSpecName "kube-api-access-m9zrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:30:19.090086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.089965    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zrc\" (UniqueName: \"kubernetes.io/projected/e4de2540-e06f-4a30-8a9f-760ecd705b55-kube-api-access-m9zrc\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:19.090086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.090004    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4de2540-e06f-4a30-8a9f-760ecd705b55-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:19.131181 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.131150    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:19.191239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.191205    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f20f65-0bca-44ed-b3ab-05e8691623e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f20f65-0bca-44ed-b3ab-05e8691623e9-maas-api-service-ca\") pod \"83f20f65-0bca-44ed-b3ab-05e8691623e9\" (UID: \"83f20f65-0bca-44ed-b3ab-05e8691623e9\") "
Nov 01 15:30:19.191446 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.191313    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f20f65-0bca-44ed-b3ab-05e8691623e9-kube-api-access-lv4zk\" (UniqueName: \"kubernetes.io/projected/83f20f65-0bca-44ed-b3ab-05e8691623e9-kube-api-access-lv4zk\") pod \"83f20f65-0bca-44ed-b3ab-05e8691623e9\" (UID: \"83f20f65-0bca-44ed-b3ab-05e8691623e9\") "
Nov 01 15:30:19.191630 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.191597    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f20f65-0bca-44ed-b3ab-05e8691623e9-maas-api-service-ca" pod "83f20f65-0bca-44ed-b3ab-05e8691623e9" (UID: "83f20f65-0bca-44ed-b3ab-05e8691623e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:30:19.191797 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.191781    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f20f65-0bca-44ed-b3ab-05e8691623e9-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:19.193637 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.193615    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f20f65-0bca-44ed-b3ab-05e8691623e9-kube-api-access-lv4zk" pod "83f20f65-0bca-44ed-b3ab-05e8691623e9" (UID: "83f20f65-0bca-44ed-b3ab-05e8691623e9"). InnerVolumeSpecName "kube-api-access-lv4zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:30:19.292582 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.292509    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4zk\" (UniqueName: \"kubernetes.io/projected/83f20f65-0bca-44ed-b3ab-05e8691623e9-kube-api-access-lv4zk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:19.483045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.483003    5423 generic.go:356] "Generic (PLEG): container finished" podID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerID="5884458ef084e665e775a492116b734dbcade9a45fa51777df218dfb44c394c1" exitCode=60
Nov 01 15:30:19.483577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.483073    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" event={"ID":"3d5493b1-07ad-40d8-896c-e87e1cf2d09a","Type":"ContainerDied","Data":"5884458ef084e665e775a492116b734dbcade9a45fa51777df218dfb44c394c1"}
Nov 01 15:30:19.483577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.483133    5423 scope.go:122] "RemoveContainer" containerID="f4023c4a83fe6a9d10257d8c9eb44a87f0745fed94bd7cac330197e4a0524065"
Nov 01 15:30:19.483577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.483548    5423 scope.go:122] "RemoveContainer" containerID="5884458ef084e665e775a492116b734dbcade9a45fa51777df218dfb44c394c1"
Nov 01 15:30:19.483895 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:19.483871    5423 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d_odh-ai-gateway-infra(3d5493b1-07ad-40d8-896c-e87e1cf2d09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a"
Nov 01 15:30:19.484536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.484519    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"
Nov 01 15:30:19.484672 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.484534    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf" event={"ID":"83f20f65-0bca-44ed-b3ab-05e8691623e9","Type":"ContainerDied","Data":"525fad7b7e494be7cd8628128a72ddad03fba88512a507763e854a9518373312"}
Nov 01 15:30:19.485985 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.485964    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879" event={"ID":"e4de2540-e06f-4a30-8a9f-760ecd705b55","Type":"ContainerDied","Data":"3a596f31d4139faa7705847112ea71d48eca89f0e9ca659d29afa04aa13091c2"}
Nov 01 15:30:19.486094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.486004    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"
Nov 01 15:30:19.497823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.497806    5423 scope.go:122] "RemoveContainer" containerID="9f165dac3a4f612cbace59055c8dcc6d2085f058ded315f289183fc5b56012fe"
Nov 01 15:30:19.506953 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.506911    5423 scope.go:122] "RemoveContainer" containerID="ac3c656ddc8747f355132b9de2d7566ab9baee3dc43696fd5190443a049f4856"
Nov 01 15:30:19.599449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.599402    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"]
Nov 01 15:30:19.733939 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.733902    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"]
Nov 01 15:30:19.736997 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.736970    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892450-n4879"]
Nov 01 15:30:19.847640 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.847608    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"]
Nov 01 15:30:19.873866 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.854179    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892450-gftrf"]
Nov 01 15:30:19.963178 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.963148    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" path="/var/lib/kubelet/pods/0bb90833-4a70-451b-8267-52dd88ea0205/volumes"
Nov 01 15:30:19.963601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.963586    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7238518f-23ad-47ec-b759-a7b32827024f" path="/var/lib/kubelet/pods/7238518f-23ad-47ec-b759-a7b32827024f/volumes"
Nov 01 15:30:19.963963 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.963951    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" path="/var/lib/kubelet/pods/83f20f65-0bca-44ed-b3ab-05e8691623e9/volumes"
Nov 01 15:30:19.964626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:19.964612    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" path="/var/lib/kubelet/pods/e4de2540-e06f-4a30-8a9f-760ecd705b55/volumes"
Nov 01 15:30:20.659078 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:20.659051    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:20.709111 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:20.709078    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-kube-api-access-zcs6d\" (UniqueName: \"kubernetes.io/projected/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-kube-api-access-zcs6d\") pod \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\" (UID: \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\") "
Nov 01 15:30:20.709288 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:20.709185    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-maas-api-service-ca\") pod \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\" (UID: \"3d5493b1-07ad-40d8-896c-e87e1cf2d09a\") "
Nov 01 15:30:20.709601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:20.709577    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-maas-api-service-ca" pod "3d5493b1-07ad-40d8-896c-e87e1cf2d09a" (UID: "3d5493b1-07ad-40d8-896c-e87e1cf2d09a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:30:20.709729 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:20.709711    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-maas-api-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:20.711222 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:20.711191    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-kube-api-access-zcs6d" pod "3d5493b1-07ad-40d8-896c-e87e1cf2d09a" (UID: "3d5493b1-07ad-40d8-896c-e87e1cf2d09a"). InnerVolumeSpecName "kube-api-access-zcs6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:30:20.811356 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:20.811323    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcs6d\" (UniqueName: \"kubernetes.io/projected/3d5493b1-07ad-40d8-896c-e87e1cf2d09a-kube-api-access-zcs6d\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:21.507030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:21.506993    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"
Nov 01 15:30:21.507260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:21.507028    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d" event={"ID":"3d5493b1-07ad-40d8-896c-e87e1cf2d09a","Type":"ContainerDied","Data":"c728952e7c09d3922448a02537b5bf7ba9d4aee27b7c1a4190b96f243b89d80b"}
Nov 01 15:30:21.507260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:21.507089    5423 scope.go:122] "RemoveContainer" containerID="5884458ef084e665e775a492116b734dbcade9a45fa51777df218dfb44c394c1"
Nov 01 15:30:21.674269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:21.674234    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"]
Nov 01 15:30:21.684413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:21.684361    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892450-5xt7d"]
Nov 01 15:30:21.961051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:21.961014    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" path="/var/lib/kubelet/pods/3d5493b1-07ad-40d8-896c-e87e1cf2d09a/volumes"
Nov 01 15:30:35.149833 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:30:35.149796    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:30:48.653263 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653220    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"]
Nov 01 15:30:48.653923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653903    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:48.653970 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653925    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:48.653970 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653936    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:48.653970 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653941    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:48.653970 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653951    5423 cpu_manager.go:414] "containerMap: removing container" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:48.653970 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653956    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:48.653970 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653971    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653976    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653987    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653991    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.653998    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654002    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654011    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654017    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654027    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654033    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654041    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654046    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654052    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654056    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654068    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654074    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654091    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654099    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654116    5423 cpu_manager.go:414] "containerMap: removing container" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:48.654197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654120    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654270    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654282    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654289    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654295    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654302    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654311    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654318    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654329    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4de2540-e06f-4a30-8a9f-760ecd705b55" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654350    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654363    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:48.655026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.654370    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:48.657503 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.657488    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.711312 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.711280    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3f45c3-openshift-default-dockercfg-tvv8g\""
Nov 01 15:30:48.740125 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740095    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-credential-socket\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.740266 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740149    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-data\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.740331 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740304    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-certs\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.740462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740447    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74a9078-63f9-4173-bbe5-e893254c0f8d-istiod-ca-cert\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.740536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740480    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-podinfo\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.740536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740517    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2db62\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-kube-api-access-2db62\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.740615 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740591    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-socket\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.740697 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740680    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-token\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.740755 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.740742    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-envoy\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.765620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.765590    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"]
Nov 01 15:30:48.842345 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842307    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-credential-socket\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.842545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842370    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-data\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.842545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842462    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-certs\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.842636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842547    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74a9078-63f9-4173-bbe5-e893254c0f8d-istiod-ca-cert\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.842636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842573    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-podinfo\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.842636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842608    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2db62\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-kube-api-access-2db62\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.842748 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842643    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-socket\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.842748 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842709    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-token\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.842845 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.842772    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-envoy\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.845213 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.845178    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-envoy\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.845485 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.845464    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-credential-socket\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.845711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.845690    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-data\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.845913 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.845896    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-certs\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.846601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.846579    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74a9078-63f9-4173-bbe5-e893254c0f8d-istiod-ca-cert\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.847716 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.847692    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-socket\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.849286 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.849263    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-podinfo\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.906408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.906307    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-token\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.906606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.906589    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2db62\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-kube-api-access-2db62\") pod \"e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") " pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:48.979328 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:48.974124    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:49.405475 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:49.405436    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"]
Nov 01 15:30:49.440896 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:49.440863    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74a9078_63f9_4173_bbe5_e893254c0f8d.slice/crio-87d2ebe5703c3b82f8c36303ecbab23d7862815a728a5d732b2ef0d31039a1fa WatchSource:0}: Error finding container 87d2ebe5703c3b82f8c36303ecbab23d7862815a728a5d732b2ef0d31039a1fa: Status 404 returned error can't find the container with id 87d2ebe5703c3b82f8c36303ecbab23d7862815a728a5d732b2ef0d31039a1fa
Nov 01 15:30:49.443510 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:49.443479    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:30:49.443617 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:49.443553    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:30:49.443617 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:49.443586    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:30:49.687348 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:49.687252    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" event={"ID":"e74a9078-63f9-4173-bbe5-e893254c0f8d","Type":"ContainerStarted","Data":"fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36"}
Nov 01 15:30:49.687348 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:49.687319    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" event={"ID":"e74a9078-63f9-4173-bbe5-e893254c0f8d","Type":"ContainerStarted","Data":"87d2ebe5703c3b82f8c36303ecbab23d7862815a728a5d732b2ef0d31039a1fa"}
Nov 01 15:30:49.808204 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:49.808154    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" podStartSLOduration=1.808139183 podStartE2EDuration="1.808139183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:49.807078913 +0000 UTC m=+4389.972564224" watchObservedRunningTime="2026-11-01 15:30:49.808139183 +0000 UTC m=+4389.973624480"
Nov 01 15:30:49.979515 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:49.979424    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:50.980884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:50.980854    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:51.698525 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:51.698498    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:51.699341 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:51.699324    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:30:53.076135 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076100    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"]
Nov 01 15:30:53.076728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076713    5423 cpu_manager.go:414] "containerMap: removing container" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:53.076774 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076729    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:53.076774 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076752    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:53.076774 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076758    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:53.076876 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076849    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f20f65-0bca-44ed-b3ab-05e8691623e9" containerName="cleanup"
Nov 01 15:30:53.076876 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076861    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5493b1-07ad-40d8-896c-e87e1cf2d09a" containerName="cleanup"
Nov 01 15:30:53.076876 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076867    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7238518f-23ad-47ec-b759-a7b32827024f" containerName="cleanup"
Nov 01 15:30:53.076876 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.076876    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb90833-4a70-451b-8267-52dd88ea0205" containerName="cleanup"
Nov 01 15:30:53.080333 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.080312    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.106815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.106793    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-envoy\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.106919 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.106842    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnlk\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-kube-api-access-qlnlk\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.107019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.106960    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istiod-ca-cert\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.107077 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.107061    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-data\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.107124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.107097    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-socket\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.107166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.107144    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-token\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.107223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.107212    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-credential-socket\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.107270 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.107235    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-podinfo\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.107270 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.107254    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-certs\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.133766 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.133735    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-600cc7-openshift-default-dockercfg-tp28v\""
Nov 01 15:30:53.188689 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.188656    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"]
Nov 01 15:30:53.208867 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.208811    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-envoy\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209081 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.208876    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnlk\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-kube-api-access-qlnlk\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209081 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.208908    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istiod-ca-cert\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209081 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.208956    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-data\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209081 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.208998    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-socket\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209081 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.209063    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-token\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209544 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.209102    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-credential-socket\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209544 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.209132    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-podinfo\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209544 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.209159    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-certs\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.209782 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.209652    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-certs\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.210230 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.210187    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-data\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.210738 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.210712    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-credential-socket\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.211231 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.211210    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istiod-ca-cert\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.211765 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.211743    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-socket\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.228337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.215589    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-podinfo\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.247530 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.228714    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-envoy\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.342681 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.332971    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-token\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.342681 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.333687    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnlk\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-kube-api-access-qlnlk\") pod \"e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") " pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.398590 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.396859    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:53.832834 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:53.832780    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67c9c20_ffd7_4b60_89ce_2a6c7868a5e0.slice/crio-3a1a76accb09982c2b07e03359a3bf43bd34dc2fc38b3e7ee01a2f7a9f788842 WatchSource:0}: Error finding container 3a1a76accb09982c2b07e03359a3bf43bd34dc2fc38b3e7ee01a2f7a9f788842: Status 404 returned error can't find the container with id 3a1a76accb09982c2b07e03359a3bf43bd34dc2fc38b3e7ee01a2f7a9f788842
Nov 01 15:30:53.835781 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.835738    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:30:53.835893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.835819    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:30:53.835893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.835865    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:30:53.836587 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:53.836565    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"]
Nov 01 15:30:54.719751 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:54.719715    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" event={"ID":"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0","Type":"ContainerStarted","Data":"29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4"}
Nov 01 15:30:54.720160 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:54.719771    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" event={"ID":"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0","Type":"ContainerStarted","Data":"3a1a76accb09982c2b07e03359a3bf43bd34dc2fc38b3e7ee01a2f7a9f788842"}
Nov 01 15:30:54.890314 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:54.890263    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" podStartSLOduration=1.890249495 podStartE2EDuration="1.890249495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:54.889883657 +0000 UTC m=+4395.055368968" watchObservedRunningTime="2026-11-01 15:30:54.890249495 +0000 UTC m=+4395.055734796"
Nov 01 15:30:55.397576 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:55.397544    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:55.403453 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:55.403430    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:55.726037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:55.725960    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:55.726823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:55.726803    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:30:56.362512 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.362478    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"]
Nov 01 15:30:56.366700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.366680    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:56.438412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.434196    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3f45c3-dockercfg-m7bjj\""
Nov 01 15:30:56.438412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.435004    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3f45c3\""
Nov 01 15:30:56.438412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.435601    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"]
Nov 01 15:30:56.479283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.479025    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fct82\" (UniqueName: \"kubernetes.io/projected/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-kube-api-access-fct82\") pod \"payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b\" (UID: \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:56.481561 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.481486    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b\" (UID: \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:56.543963 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.543902    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"]
Nov 01 15:30:56.549698 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.549667    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:56.583589 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.583554    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fct82\" (UniqueName: \"kubernetes.io/projected/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-kube-api-access-fct82\") pod \"payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b\" (UID: \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:56.583743 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.583639    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b\" (UID: \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:56.584337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.584316    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b\" (UID: \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:56.594101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.594074    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"]
Nov 01 15:30:56.642256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.642184    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fct82\" (UniqueName: \"kubernetes.io/projected/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-kube-api-access-fct82\") pod \"payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b\" (UID: \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:56.683573 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.683542    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:56.684972 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.684952    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727d2dcc-9e55-4a28-8f6b-85510fb014fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv\" (UID: \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:56.685181 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.685165    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bzlc\" (UniqueName: \"kubernetes.io/projected/727d2dcc-9e55-4a28-8f6b-85510fb014fd-kube-api-access-6bzlc\") pod \"payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv\" (UID: \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:56.787022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.786984    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727d2dcc-9e55-4a28-8f6b-85510fb014fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv\" (UID: \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:56.787491 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.787235    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzlc\" (UniqueName: \"kubernetes.io/projected/727d2dcc-9e55-4a28-8f6b-85510fb014fd-kube-api-access-6bzlc\") pod \"payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv\" (UID: \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:56.787832 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.787810    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727d2dcc-9e55-4a28-8f6b-85510fb014fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv\" (UID: \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:56.845508 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.845481    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzlc\" (UniqueName: \"kubernetes.io/projected/727d2dcc-9e55-4a28-8f6b-85510fb014fd-kube-api-access-6bzlc\") pod \"payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv\" (UID: \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:56.864526 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:56.864489    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:57.057433 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:57.057354    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a0e0fa_f4e9_49d3_abcf_67c6cd089143.slice/crio-6634769ae14263bf0850372194d7a47a5fea1f56e4d1ed720ac66e3fa62d4a9c WatchSource:0}: Error finding container 6634769ae14263bf0850372194d7a47a5fea1f56e4d1ed720ac66e3fa62d4a9c: Status 404 returned error can't find the container with id 6634769ae14263bf0850372194d7a47a5fea1f56e4d1ed720ac66e3fa62d4a9c
Nov 01 15:30:57.060930 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.060908    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"]
Nov 01 15:30:57.233344 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:30:57.233303    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727d2dcc_9e55_4a28_8f6b_85510fb014fd.slice/crio-e62cde35760ba02ed560aa4af996019585d7de355762635b58145589ef370c73 WatchSource:0}: Error finding container e62cde35760ba02ed560aa4af996019585d7de355762635b58145589ef370c73: Status 404 returned error can't find the container with id e62cde35760ba02ed560aa4af996019585d7de355762635b58145589ef370c73
Nov 01 15:30:57.236149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.236126    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"]
Nov 01 15:30:57.741038 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.740996    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b" event={"ID":"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143","Type":"ContainerStarted","Data":"beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c"}
Nov 01 15:30:57.741232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.741057    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b" event={"ID":"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143","Type":"ContainerStarted","Data":"6634769ae14263bf0850372194d7a47a5fea1f56e4d1ed720ac66e3fa62d4a9c"}
Nov 01 15:30:57.741232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.741084    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:30:57.742786 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.742754    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv" event={"ID":"727d2dcc-9e55-4a28-8f6b-85510fb014fd","Type":"ContainerStarted","Data":"2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28"}
Nov 01 15:30:57.742921 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.742803    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv" event={"ID":"727d2dcc-9e55-4a28-8f6b-85510fb014fd","Type":"ContainerStarted","Data":"e62cde35760ba02ed560aa4af996019585d7de355762635b58145589ef370c73"}
Nov 01 15:30:57.742975 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.742928    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:30:57.860260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.860203    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b" podStartSLOduration=1.860191496 podStartE2EDuration="1.860191496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:57.858667374 +0000 UTC m=+4398.024152677" watchObservedRunningTime="2026-11-01 15:30:57.860191496 +0000 UTC m=+4398.025676794"
Nov 01 15:30:57.997311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:30:57.997200    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv" podStartSLOduration=1.997184603 podStartE2EDuration="1.997184603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:57.994243081 +0000 UTC m=+4398.159728380" watchObservedRunningTime="2026-11-01 15:30:57.997184603 +0000 UTC m=+4398.162669901"
Nov 01 15:31:03.091304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.091265    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"]
Nov 01 15:31:03.095823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.095799    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:03.153456 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.153420    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"]
Nov 01 15:31:03.190409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.188550    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342fd0eb-3d70-4109-8784-5addd7934bfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf\" (UID: \"342fd0eb-3d70-4109-8784-5addd7934bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:03.190409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.188721    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvx78\" (UniqueName: \"kubernetes.io/projected/342fd0eb-3d70-4109-8784-5addd7934bfb-kube-api-access-gvx78\") pod \"payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf\" (UID: \"342fd0eb-3d70-4109-8784-5addd7934bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:03.250421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.238172    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-600cc7-dockercfg-b9tdc\""
Nov 01 15:31:03.254413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.243363    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-600cc7\""
Nov 01 15:31:03.292506 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.292332    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342fd0eb-3d70-4109-8784-5addd7934bfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf\" (UID: \"342fd0eb-3d70-4109-8784-5addd7934bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:03.292716 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.292510    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvx78\" (UniqueName: \"kubernetes.io/projected/342fd0eb-3d70-4109-8784-5addd7934bfb-kube-api-access-gvx78\") pod \"payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf\" (UID: \"342fd0eb-3d70-4109-8784-5addd7934bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:03.293622 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.293595    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342fd0eb-3d70-4109-8784-5addd7934bfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf\" (UID: \"342fd0eb-3d70-4109-8784-5addd7934bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:03.320911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.319426    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"]
Nov 01 15:31:03.325041 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.325021    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"]
Nov 01 15:31:03.325258 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.325247    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:03.350903 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.350842    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvx78\" (UniqueName: \"kubernetes.io/projected/342fd0eb-3d70-4109-8784-5addd7934bfb-kube-api-access-gvx78\") pod \"payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf\" (UID: \"342fd0eb-3d70-4109-8784-5addd7934bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:03.393661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.393633    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hc8\" (UniqueName: \"kubernetes.io/projected/1c9692a3-6f28-4c34-b73d-297815b3845b-kube-api-access-n8hc8\") pod \"payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526\" (UID: \"1c9692a3-6f28-4c34-b73d-297815b3845b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:03.393829 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.393817    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9692a3-6f28-4c34-b73d-297815b3845b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526\" (UID: \"1c9692a3-6f28-4c34-b73d-297815b3845b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:03.410537 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.410512    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:03.495518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.495471    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9692a3-6f28-4c34-b73d-297815b3845b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526\" (UID: \"1c9692a3-6f28-4c34-b73d-297815b3845b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:03.495662 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.495565    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hc8\" (UniqueName: \"kubernetes.io/projected/1c9692a3-6f28-4c34-b73d-297815b3845b-kube-api-access-n8hc8\") pod \"payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526\" (UID: \"1c9692a3-6f28-4c34-b73d-297815b3845b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:03.496294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.496260    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9692a3-6f28-4c34-b73d-297815b3845b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526\" (UID: \"1c9692a3-6f28-4c34-b73d-297815b3845b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:03.553415 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.553372    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hc8\" (UniqueName: \"kubernetes.io/projected/1c9692a3-6f28-4c34-b73d-297815b3845b-kube-api-access-n8hc8\") pod \"payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526\" (UID: \"1c9692a3-6f28-4c34-b73d-297815b3845b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:03.647901 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.647825    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:03.783653 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:31:03.779719    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342fd0eb_3d70_4109_8784_5addd7934bfb.slice/crio-d7dbb063609d113eaa652274cdf96242baf84f95e7f4de816dfe985e30dc4b05 WatchSource:0}: Error finding container d7dbb063609d113eaa652274cdf96242baf84f95e7f4de816dfe985e30dc4b05: Status 404 returned error can't find the container with id d7dbb063609d113eaa652274cdf96242baf84f95e7f4de816dfe985e30dc4b05
Nov 01 15:31:03.783653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:03.783533    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"]
Nov 01 15:31:04.026638 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:31:04.026601    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9692a3_6f28_4c34_b73d_297815b3845b.slice/crio-ec55071671e7ca6f7de87d684c8acecba0503787f84f919e8664a1713f601903 WatchSource:0}: Error finding container ec55071671e7ca6f7de87d684c8acecba0503787f84f919e8664a1713f601903: Status 404 returned error can't find the container with id ec55071671e7ca6f7de87d684c8acecba0503787f84f919e8664a1713f601903
Nov 01 15:31:04.029649 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:04.029623    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"]
Nov 01 15:31:04.788752 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:04.788722    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf" event={"ID":"342fd0eb-3d70-4109-8784-5addd7934bfb","Type":"ContainerStarted","Data":"f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb"}
Nov 01 15:31:04.789209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:04.788774    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf" event={"ID":"342fd0eb-3d70-4109-8784-5addd7934bfb","Type":"ContainerStarted","Data":"d7dbb063609d113eaa652274cdf96242baf84f95e7f4de816dfe985e30dc4b05"}
Nov 01 15:31:04.789209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:04.788937    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:04.790690 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:04.790666    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526" event={"ID":"1c9692a3-6f28-4c34-b73d-297815b3845b","Type":"ContainerStarted","Data":"efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83"}
Nov 01 15:31:04.790823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:04.790711    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526" event={"ID":"1c9692a3-6f28-4c34-b73d-297815b3845b","Type":"ContainerStarted","Data":"ec55071671e7ca6f7de87d684c8acecba0503787f84f919e8664a1713f601903"}
Nov 01 15:31:04.790823 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:04.790746    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:04.909787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:04.909736    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf" podStartSLOduration=1.909723346 podStartE2EDuration="1.909723346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:31:04.909011764 +0000 UTC m=+4405.074497063" watchObservedRunningTime="2026-11-01 15:31:04.909723346 +0000 UTC m=+4405.075208641"
Nov 01 15:31:05.051491 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:05.046087    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526" podStartSLOduration=2.046064543 podStartE2EDuration="2.046064543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:31:05.032265604 +0000 UTC m=+4405.197750904" watchObservedRunningTime="2026-11-01 15:31:05.046064543 +0000 UTC m=+4405.211549842"
Nov 01 15:31:05.175041 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:31:05.174965    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:31:08.751847 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:08.751813    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:31:08.752454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:08.751899    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:31:15.801756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:15.801719    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:31:15.802308 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:15.801808    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:31:35.149488 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:31:35.149405    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:31:46.633410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:46.631494    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:31:46.633410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:46.631576    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:31:46.977602 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:46.977501    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:31:46.977602 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:31:46.977564    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:32:05.149443 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:32:05.149405    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:32:24.133620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:24.133577    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:32:34.401561 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:34.401520    5423 generic.go:356] "Generic (PLEG): container finished" podID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerID="fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36" exitCode=0
Nov 01 15:32:34.402047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:34.401596    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" event={"ID":"e74a9078-63f9-4173-bbe5-e893254c0f8d","Type":"ContainerDied","Data":"fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36"}
Nov 01 15:32:34.402047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:34.402015    5423 scope.go:122] "RemoveContainer" containerID="fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36"
Nov 01 15:32:34.403730 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:34.403706    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:32:34.403808 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:34.403780    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:32:34.403848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:34.403814    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:32:35.149945 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:32:35.149902    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:32:35.410003 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:35.409908    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" event={"ID":"e74a9078-63f9-4173-bbe5-e893254c0f8d","Type":"ContainerStarted","Data":"2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca"}
Nov 01 15:32:35.974979 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:35.974953    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:32:35.976550 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:35.976516    5423 patch_prober.go:29] interesting pod/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Nov 01 15:32:35.976665 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:35.976586    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Nov 01 15:32:36.713412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:36.707472    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:32:36.975261 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:36.975172    5423 patch_prober.go:29] interesting pod/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Nov 01 15:32:36.975261 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:36.975238    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Nov 01 15:32:37.975588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:37.975554    5423 patch_prober.go:29] interesting pod/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Nov 01 15:32:37.976062 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:37.975605    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Nov 01 15:32:38.975660 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:38.975625    5423 patch_prober.go:29] interesting pod/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Nov 01 15:32:38.976086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:38.975680    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Nov 01 15:32:39.975218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:39.975184    5423 patch_prober.go:29] interesting pod/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Nov 01 15:32:39.975413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:39.975230    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Nov 01 15:32:40.974938 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:40.974902    5423 patch_prober.go:29] interesting pod/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Nov 01 15:32:40.975367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:40.974952    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Nov 01 15:32:41.980223 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:41.980197    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:32:41.981221 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:41.981203    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:32:43.105492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:43.105453    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:32:43.137711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:43.137689    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:32:50.161162 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.161084    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"]
Nov 01 15:32:50.164794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.164778    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.211469    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f762aa0-047b-40cd-8dd5-d179e8235f44-istiod-ca-cert\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.211519    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsln\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-kube-api-access-8wsln\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.211641    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-certs\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.211702    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-credential-socket\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.211736    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-socket\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.211939    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-data\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.211971    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-token\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.212298    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-podinfo\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.215482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.212410    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-envoy\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.249959 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.249926    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6a0fa146-openshift-default-dockercfg-s5w8q\""
Nov 01 15:32:50.320452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.320413    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-credential-socket\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.320726 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.320703    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-socket\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.320997 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.320976    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-data\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.321142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.321127    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-token\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.321278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.321265    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-podinfo\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.321524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.321507    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-envoy\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.321652 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.321636    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f762aa0-047b-40cd-8dd5-d179e8235f44-istiod-ca-cert\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.321756 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.321741    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsln\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-kube-api-access-8wsln\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.321927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.321913    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-certs\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.322532 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.322509    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-certs\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.325329 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.325306    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-credential-socket\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.325775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.325753    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-socket\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.326150 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.326129    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-data\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.327565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.327540    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f762aa0-047b-40cd-8dd5-d179e8235f44-istiod-ca-cert\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.373536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.373494    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"]
Nov 01 15:32:50.408373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.387526    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-token\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.408373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.390631    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsln\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-kube-api-access-8wsln\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.423312 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.423193    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-envoy\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.432501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.424220    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-podinfo\") pod \"e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") " pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.477043 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.477017    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:50.925705 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.925671    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"]
Nov 01 15:32:50.958785 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:32:50.958758    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f762aa0_047b_40cd_8dd5_d179e8235f44.slice/crio-7559c4fc88c9264cbe669cd8a226f09f687e26ef6b50982144d0dc5aa673047a WatchSource:0}: Error finding container 7559c4fc88c9264cbe669cd8a226f09f687e26ef6b50982144d0dc5aa673047a: Status 404 returned error can't find the container with id 7559c4fc88c9264cbe669cd8a226f09f687e26ef6b50982144d0dc5aa673047a
Nov 01 15:32:50.961260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.961225    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:32:50.961334 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.961294    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:32:50.961334 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:50.961326    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:32:51.552087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:51.552049    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" event={"ID":"9f762aa0-047b-40cd-8dd5-d179e8235f44","Type":"ContainerStarted","Data":"c6b1157ca5c22f7ec2b6b0a20e3e934a784ea5382941ca637d9f6b02ae4e5fbd"}
Nov 01 15:32:51.552653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:51.552113    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" event={"ID":"9f762aa0-047b-40cd-8dd5-d179e8235f44","Type":"ContainerStarted","Data":"7559c4fc88c9264cbe669cd8a226f09f687e26ef6b50982144d0dc5aa673047a"}
Nov 01 15:32:51.671833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:51.671780    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" podStartSLOduration=1.671767548 podStartE2EDuration="1.671767548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:32:51.670768584 +0000 UTC m=+4511.836253889" watchObservedRunningTime="2026-11-01 15:32:51.671767548 +0000 UTC m=+4511.837252846"
Nov 01 15:32:52.477560 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:52.477522    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:52.483335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:52.483314    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:52.558092 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:52.558065    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:52.558942 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:52.558922    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:32:55.370636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:55.370599    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:32:58.043849 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.043821    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"]
Nov 01 15:32:58.048779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.048754    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:58.126025 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.125987    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"]
Nov 01 15:32:58.169101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.168894    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6a0fa146-dockercfg-6wgvh\""
Nov 01 15:32:58.169101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.168979    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6a0fa146\""
Nov 01 15:32:58.174415 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.170600    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9cv\" (UniqueName: \"kubernetes.io/projected/6f4ee975-5cff-493d-8107-3bc0039b01f9-kube-api-access-hx9cv\") pod \"payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s\" (UID: \"6f4ee975-5cff-493d-8107-3bc0039b01f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:58.187219 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.186204    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4ee975-5cff-493d-8107-3bc0039b01f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s\" (UID: \"6f4ee975-5cff-493d-8107-3bc0039b01f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:58.227759 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.227717    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"]
Nov 01 15:32:58.247421 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.245698    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:58.346047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.345989    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0cbc5-a4e1-48ac-bc28-486970518262-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4\" (UID: \"6ae0cbc5-a4e1-48ac-bc28-486970518262\") " pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:58.346263 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.346074    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9cv\" (UniqueName: \"kubernetes.io/projected/6f4ee975-5cff-493d-8107-3bc0039b01f9-kube-api-access-hx9cv\") pod \"payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s\" (UID: \"6f4ee975-5cff-493d-8107-3bc0039b01f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:58.346263 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.346139    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4ee975-5cff-493d-8107-3bc0039b01f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s\" (UID: \"6f4ee975-5cff-493d-8107-3bc0039b01f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:58.346406 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.346356    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgz2\" (UniqueName: \"kubernetes.io/projected/6ae0cbc5-a4e1-48ac-bc28-486970518262-kube-api-access-6zgz2\") pod \"payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4\" (UID: \"6ae0cbc5-a4e1-48ac-bc28-486970518262\") " pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:58.360040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.347884    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4ee975-5cff-493d-8107-3bc0039b01f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s\" (UID: \"6f4ee975-5cff-493d-8107-3bc0039b01f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:58.360500 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.360275    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"]
Nov 01 15:32:58.421760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.421729    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9cv\" (UniqueName: \"kubernetes.io/projected/6f4ee975-5cff-493d-8107-3bc0039b01f9-kube-api-access-hx9cv\") pod \"payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s\" (UID: \"6f4ee975-5cff-493d-8107-3bc0039b01f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:58.448190 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.448153    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0cbc5-a4e1-48ac-bc28-486970518262-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4\" (UID: \"6ae0cbc5-a4e1-48ac-bc28-486970518262\") " pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:58.448372 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.448289    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgz2\" (UniqueName: \"kubernetes.io/projected/6ae0cbc5-a4e1-48ac-bc28-486970518262-kube-api-access-6zgz2\") pod \"payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4\" (UID: \"6ae0cbc5-a4e1-48ac-bc28-486970518262\") " pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:58.448969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.448943    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0cbc5-a4e1-48ac-bc28-486970518262-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4\" (UID: \"6ae0cbc5-a4e1-48ac-bc28-486970518262\") " pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:58.521306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.521272    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgz2\" (UniqueName: \"kubernetes.io/projected/6ae0cbc5-a4e1-48ac-bc28-486970518262-kube-api-access-6zgz2\") pod \"payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4\" (UID: \"6ae0cbc5-a4e1-48ac-bc28-486970518262\") " pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:58.664043 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.663960    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:58.677942 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:58.677916    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:59.066908 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:32:59.066875    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4ee975_5cff_493d_8107_3bc0039b01f9.slice/crio-4411791643c125a6dd136d9832c736eff32fd19aadfef1d69dd4ad1463c59af0 WatchSource:0}: Error finding container 4411791643c125a6dd136d9832c736eff32fd19aadfef1d69dd4ad1463c59af0: Status 404 returned error can't find the container with id 4411791643c125a6dd136d9832c736eff32fd19aadfef1d69dd4ad1463c59af0
Nov 01 15:32:59.070723 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.070697    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"]
Nov 01 15:32:59.124423 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:32:59.124362    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae0cbc5_a4e1_48ac_bc28_486970518262.slice/crio-c77ef8f7e3dacd27a1d5a20896048866a17b0fc8f2c90e2f6b2813943f6b677d WatchSource:0}: Error finding container c77ef8f7e3dacd27a1d5a20896048866a17b0fc8f2c90e2f6b2813943f6b677d: Status 404 returned error can't find the container with id c77ef8f7e3dacd27a1d5a20896048866a17b0fc8f2c90e2f6b2813943f6b677d
Nov 01 15:32:59.124595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.124568    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"]
Nov 01 15:32:59.643491 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.643452    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4" event={"ID":"6ae0cbc5-a4e1-48ac-bc28-486970518262","Type":"ContainerStarted","Data":"7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d"}
Nov 01 15:32:59.643711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.643511    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4" event={"ID":"6ae0cbc5-a4e1-48ac-bc28-486970518262","Type":"ContainerStarted","Data":"c77ef8f7e3dacd27a1d5a20896048866a17b0fc8f2c90e2f6b2813943f6b677d"}
Nov 01 15:32:59.643711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.643560    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:32:59.645158 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.645136    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s" event={"ID":"6f4ee975-5cff-493d-8107-3bc0039b01f9","Type":"ContainerStarted","Data":"b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a"}
Nov 01 15:32:59.645267 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.645175    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s" event={"ID":"6f4ee975-5cff-493d-8107-3bc0039b01f9","Type":"ContainerStarted","Data":"4411791643c125a6dd136d9832c736eff32fd19aadfef1d69dd4ad1463c59af0"}
Nov 01 15:32:59.645267 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.645225    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:32:59.767090 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.767039    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4" podStartSLOduration=1.7670256690000001 podStartE2EDuration="1.767025669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:32:59.764334887 +0000 UTC m=+4519.929820186" watchObservedRunningTime="2026-11-01 15:32:59.767025669 +0000 UTC m=+4519.932510966"
Nov 01 15:32:59.891131 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:32:59.887315    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s" podStartSLOduration=2.887289183 podStartE2EDuration="2.887289183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:32:59.885813139 +0000 UTC m=+4520.051298443" watchObservedRunningTime="2026-11-01 15:32:59.887289183 +0000 UTC m=+4520.052774510"
Nov 01 15:33:05.150219 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:33:05.150176    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:33:06.693586 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:06.693553    5423 generic.go:356] "Generic (PLEG): container finished" podID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerID="29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4" exitCode=0
Nov 01 15:33:06.694046 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:06.693622    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" event={"ID":"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0","Type":"ContainerDied","Data":"29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4"}
Nov 01 15:33:06.694046 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:06.693999    5423 scope.go:122] "RemoveContainer" containerID="29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4"
Nov 01 15:33:06.695582 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:06.695556    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:33:06.695661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:06.695619    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:33:06.695661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:06.695649    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:33:07.714410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:07.707847    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" event={"ID":"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0","Type":"ContainerStarted","Data":"14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1"}
Nov 01 15:33:08.419412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:08.416445    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:33:08.419630 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:08.419527    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 01 15:33:08.419630 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:08.419594    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 01 15:33:09.413410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:09.412737    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 01 15:33:09.413410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:09.412813    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 01 15:33:10.421409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:10.420490    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 01 15:33:10.421409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:10.420552    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 01 15:33:10.654692 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:10.654663    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:33:10.654880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:10.654729    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:33:10.725948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:10.725864    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:33:11.407410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:11.407310    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 01 15:33:11.407410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:11.407373    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 01 15:33:12.411408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:12.402698    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Nov 01 15:33:12.411408 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:12.402797    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Nov 01 15:33:13.401891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:13.401859    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:33:13.402883 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:13.402865    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:33:31.312890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:33:31.312854    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:33:35.151362 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:33:35.151318    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:34:05.150014 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:34:05.149971    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:34:34.825192 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:34.825114    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:34:35.150379 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:34:35.150279    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:34:45.657233 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:45.657198    5423 generic.go:356] "Generic (PLEG): container finished" podID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerID="c6b1157ca5c22f7ec2b6b0a20e3e934a784ea5382941ca637d9f6b02ae4e5fbd" exitCode=0
Nov 01 15:34:45.657724 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:45.657257    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" event={"ID":"9f762aa0-047b-40cd-8dd5-d179e8235f44","Type":"ContainerDied","Data":"c6b1157ca5c22f7ec2b6b0a20e3e934a784ea5382941ca637d9f6b02ae4e5fbd"}
Nov 01 15:34:45.657724 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:45.657675    5423 scope.go:122] "RemoveContainer" containerID="c6b1157ca5c22f7ec2b6b0a20e3e934a784ea5382941ca637d9f6b02ae4e5fbd"
Nov 01 15:34:45.659379 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:45.659348    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:34:45.659470 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:45.659446    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:34:45.659514 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:45.659481    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:34:46.666703 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:46.666671    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" event={"ID":"9f762aa0-047b-40cd-8dd5-d179e8235f44","Type":"ContainerStarted","Data":"6e3484d6d743af21c1f559a3ac0f6f24f7d2adb96ee318869ff614fc6047f134"}
Nov 01 15:34:47.477734 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:47.477701    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:34:47.477996 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:47.477959    5423 patch_prober.go:29] interesting pod/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 15:34:47.478042 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:47.478018    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 15:34:48.477724 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:48.477689    5423 patch_prober.go:29] interesting pod/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 15:34:48.478137 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:48.477750    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 15:34:49.477403 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:49.477347    5423 patch_prober.go:29] interesting pod/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 15:34:49.477587 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:49.477427    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 15:34:50.478336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:50.478299    5423 patch_prober.go:29] interesting pod/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 15:34:50.478760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:50.478355    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 15:34:51.478099 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:51.478061    5423 patch_prober.go:29] interesting pod/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 15:34:51.478303 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:51.478117    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 15:34:52.482074 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:52.482042    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:34:52.482607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:52.482490    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:34:52.483220 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:34:52.483198    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:35:00.480712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.480680    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"]
Nov 01 15:35:00.483904 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.483883    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.559867 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.559840    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ca0f9e\"/\"kube-root-ca.crt\""
Nov 01 15:35:00.559867 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.559840    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ca0f9e\"/\"openshift-service-ca.crt\""
Nov 01 15:35:00.587761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.587726    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-dshm\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.587944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.587784    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05eced9e-3508-482d-9f58-90f9e0e16546-tls-certs\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.587944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.587908    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-tmp-dir\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.588031 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.587971    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-model-cache\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.588083 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.588031    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-home\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.588161 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.588136    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vphx\" (UniqueName: \"kubernetes.io/projected/05eced9e-3508-482d-9f58-90f9e0e16546-kube-api-access-5vphx\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.588750 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.588728    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ca0f9e\"/\"sub-test-model-ca0f9e-kserve-self-signed-certs\""
Nov 01 15:35:00.588862 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.588824    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ca0f9e\"/\"default-dockercfg-p4lz5\""
Nov 01 15:35:00.598903 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.597121    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"]
Nov 01 15:35:00.691073 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.691018    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-dshm\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.691724 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.691692    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05eced9e-3508-482d-9f58-90f9e0e16546-tls-certs\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.691865 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.691760    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-tmp-dir\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.691865 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.691841    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-model-cache\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.691969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.691870    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-home\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.691969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.691956    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vphx\" (UniqueName: \"kubernetes.io/projected/05eced9e-3508-482d-9f58-90f9e0e16546-kube-api-access-5vphx\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.692459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.692431    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-tmp-dir\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.692623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.692492    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-model-cache\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.692725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.692646    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-home\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.694888 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.694864    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05eced9e-3508-482d-9f58-90f9e0e16546-tls-certs\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.697352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.697332    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-dshm\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.751186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.751123    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vphx\" (UniqueName: \"kubernetes.io/projected/05eced9e-3508-482d-9f58-90f9e0e16546-kube-api-access-5vphx\") pod \"sub-test-model-ca0f9e-kserve-546858d969-m2hnw\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") " pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:00.794868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:00.794838    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:01.165331 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:35:01.165033    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05eced9e_3508_482d_9f58_90f9e0e16546.slice/crio-0b65806c1ec4528e03189dc21402f6ed92810f3a516eb382b8728debb68ebe7c WatchSource:0}: Error finding container 0b65806c1ec4528e03189dc21402f6ed92810f3a516eb382b8728debb68ebe7c: Status 404 returned error can't find the container with id 0b65806c1ec4528e03189dc21402f6ed92810f3a516eb382b8728debb68ebe7c
Nov 01 15:35:01.168628 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:01.168603    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"]
Nov 01 15:35:01.765693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:01.765656    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw" event={"ID":"05eced9e-3508-482d-9f58-90f9e0e16546","Type":"ContainerStarted","Data":"47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987"}
Nov 01 15:35:01.766070 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:01.765704    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw" event={"ID":"05eced9e-3508-482d-9f58-90f9e0e16546","Type":"ContainerStarted","Data":"0b65806c1ec4528e03189dc21402f6ed92810f3a516eb382b8728debb68ebe7c"}
Nov 01 15:35:01.805406 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:01.797475    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:01.884965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:01.884909    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw" podStartSLOduration=1.8848952190000001 podStartE2EDuration="1.884895219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:35:01.883155757 +0000 UTC m=+4642.048641056" watchObservedRunningTime="2026-11-01 15:35:01.884895219 +0000 UTC m=+4642.050380515"
Nov 01 15:35:05.149869 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:35:05.149834    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:35:08.785260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:08.785228    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:35:10.795231 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:10.795190    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:10.808322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:10.808295    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:10.845492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:10.845464    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:35:23.713311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.713272    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"]
Nov 01 15:35:23.717565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.717547    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.774882 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.774856    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"]
Nov 01 15:35:23.822061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.822031    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-974865-serving-cert\""
Nov 01 15:35:23.824420 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.824395    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.824495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.824476    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1dec20-0f8a-4623-8f6f-683fd454ede4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.824559 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.824532    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.824612 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.824599    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrjm\" (UniqueName: \"kubernetes.io/projected/bd1dec20-0f8a-4623-8f6f-683fd454ede4-kube-api-access-qwrjm\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.926225 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.926195    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrjm\" (UniqueName: \"kubernetes.io/projected/bd1dec20-0f8a-4623-8f6f-683fd454ede4-kube-api-access-qwrjm\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.926424 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.926353    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.926490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.926427    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1dec20-0f8a-4623-8f6f-683fd454ede4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.926490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.926454    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.926595 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:35:23.926507    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-serving-cert: secret "maas-api-e2e-shared-b-w0-974865-serving-cert" not found
Nov 01 15:35:23.926595 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:35:23.926589    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls podName:bd1dec20-0f8a-4623-8f6f-683fd454ede4 nodeName:}" failed. No retries permitted until 2026-11-01 15:35:24.426573223 +0000 UTC m=+4664.592058511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls") pod "maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n" (UID: "bd1dec20-0f8a-4623-8f6f-683fd454ede4") : secret "maas-api-e2e-shared-b-w0-974865-serving-cert" not found
Nov 01 15:35:23.927103 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.927080    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1dec20-0f8a-4623-8f6f-683fd454ede4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.929001 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.928980    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:23.994410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:23.994297    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrjm\" (UniqueName: \"kubernetes.io/projected/bd1dec20-0f8a-4623-8f6f-683fd454ede4-kube-api-access-qwrjm\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:24.433509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:24.433477    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:24.435906 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:24.435878    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:24.632200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:24.632163    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:25.048240 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:35:25.048206    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1dec20_0f8a_4623_8f6f_683fd454ede4.slice/crio-6c024f641007b5d28fc4ddf8f7fa56f1fbd8fbe345c7c99497e5e242ac9607a4 WatchSource:0}: Error finding container 6c024f641007b5d28fc4ddf8f7fa56f1fbd8fbe345c7c99497e5e242ac9607a4: Status 404 returned error can't find the container with id 6c024f641007b5d28fc4ddf8f7fa56f1fbd8fbe345c7c99497e5e242ac9607a4
Nov 01 15:35:25.050102 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:25.050081    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:35:25.096149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:25.096121    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"]
Nov 01 15:35:25.932319 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:25.932285    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n" event={"ID":"bd1dec20-0f8a-4623-8f6f-683fd454ede4","Type":"ContainerStarted","Data":"6c024f641007b5d28fc4ddf8f7fa56f1fbd8fbe345c7c99497e5e242ac9607a4"}
Nov 01 15:35:26.940333 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:26.940284    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n" event={"ID":"bd1dec20-0f8a-4623-8f6f-683fd454ede4","Type":"ContainerStarted","Data":"cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc"}
Nov 01 15:35:26.940841 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:26.940360    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:27.058479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:27.058338    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n" podStartSLOduration=3.083110999 podStartE2EDuration="4.058321935s" totalImagesPullingTime="975.210936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:35:27.056502447 +0000 UTC m=+4667.221987766" watchObservedRunningTime="2026-11-01 15:35:27.058321935 +0000 UTC m=+4667.223807222"
Nov 01 15:35:32.959980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:32.959946    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:35:35.149923 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:35:35.149891    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:35:56.159525 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:35:56.159447    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:36:05.150306 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:36:05.150262    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:36:05.799718 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:05.799684    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"]
Nov 01 15:36:05.908015 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:05.907980    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"]
Nov 01 15:36:10.800244 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:10.800200    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4" podUID="6ae0cbc5-a4e1-48ac-bc28-486970518262" containerName="payload-processing" containerID="cri-o://7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d" gracePeriod=25
Nov 01 15:36:10.908899 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:10.908859    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s" podUID="6f4ee975-5cff-493d-8107-3bc0039b01f9" containerName="payload-pre-processing" containerID="cri-o://b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a" gracePeriod=25
Nov 01 15:36:11.083271 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.083248    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:36:11.196956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.196927    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:36:11.207233 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.207203    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae0cbc5-a4e1-48ac-bc28-486970518262-kube-api-access-6zgz2\" (UniqueName: \"kubernetes.io/projected/6ae0cbc5-a4e1-48ac-bc28-486970518262-kube-api-access-6zgz2\") pod \"6ae0cbc5-a4e1-48ac-bc28-486970518262\" (UID: \"6ae0cbc5-a4e1-48ac-bc28-486970518262\") "
Nov 01 15:36:11.207444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.207423    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae0cbc5-a4e1-48ac-bc28-486970518262-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0cbc5-a4e1-48ac-bc28-486970518262-plugins-config-volume\") pod \"6ae0cbc5-a4e1-48ac-bc28-486970518262\" (UID: \"6ae0cbc5-a4e1-48ac-bc28-486970518262\") "
Nov 01 15:36:11.207835 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.207809    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae0cbc5-a4e1-48ac-bc28-486970518262-plugins-config-volume" pod "6ae0cbc5-a4e1-48ac-bc28-486970518262" (UID: "6ae0cbc5-a4e1-48ac-bc28-486970518262"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:36:11.207973 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.207955    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae0cbc5-a4e1-48ac-bc28-486970518262-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:36:11.209727 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.209702    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae0cbc5-a4e1-48ac-bc28-486970518262-kube-api-access-6zgz2" pod "6ae0cbc5-a4e1-48ac-bc28-486970518262" (UID: "6ae0cbc5-a4e1-48ac-bc28-486970518262"). InnerVolumeSpecName "kube-api-access-6zgz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:36:11.273685 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.273647    5423 generic.go:356] "Generic (PLEG): container finished" podID="6f4ee975-5cff-493d-8107-3bc0039b01f9" containerID="b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a" exitCode=0
Nov 01 15:36:11.273844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.273698    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"
Nov 01 15:36:11.273844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.273732    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s" event={"ID":"6f4ee975-5cff-493d-8107-3bc0039b01f9","Type":"ContainerDied","Data":"b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a"}
Nov 01 15:36:11.273844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.273808    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s" event={"ID":"6f4ee975-5cff-493d-8107-3bc0039b01f9","Type":"ContainerDied","Data":"4411791643c125a6dd136d9832c736eff32fd19aadfef1d69dd4ad1463c59af0"}
Nov 01 15:36:11.273844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.273833    5423 scope.go:122] "RemoveContainer" containerID="b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a"
Nov 01 15:36:11.275148 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.275125    5423 generic.go:356] "Generic (PLEG): container finished" podID="6ae0cbc5-a4e1-48ac-bc28-486970518262" containerID="7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d" exitCode=0
Nov 01 15:36:11.275252 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.275156    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4" event={"ID":"6ae0cbc5-a4e1-48ac-bc28-486970518262","Type":"ContainerDied","Data":"7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d"}
Nov 01 15:36:11.275252 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.275188    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"
Nov 01 15:36:11.275356 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.275197    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4" event={"ID":"6ae0cbc5-a4e1-48ac-bc28-486970518262","Type":"ContainerDied","Data":"c77ef8f7e3dacd27a1d5a20896048866a17b0fc8f2c90e2f6b2813943f6b677d"}
Nov 01 15:36:11.284495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.284480    5423 scope.go:122] "RemoveContainer" containerID="b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a"
Nov 01 15:36:11.284745 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:36:11.284728    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a\": container with ID starting with b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a not found: ID does not exist" containerID="b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a"
Nov 01 15:36:11.284792 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.284752    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a"} err="failed to get container status \"b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a\": rpc error: code = NotFound desc = could not find container \"b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a\": container with ID starting with b55dcbf906f260e38717814c3c279cf76141b48e18e9941720de7b635c1be83a not found: ID does not exist"
Nov 01 15:36:11.284792 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.284768    5423 scope.go:122] "RemoveContainer" containerID="7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d"
Nov 01 15:36:11.294036 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.294020    5423 scope.go:122] "RemoveContainer" containerID="7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d"
Nov 01 15:36:11.294276 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:36:11.294258    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d\": container with ID starting with 7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d not found: ID does not exist" containerID="7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d"
Nov 01 15:36:11.294339 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.294285    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d"} err="failed to get container status \"7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d\": rpc error: code = NotFound desc = could not find container \"7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d\": container with ID starting with 7dae1182432afcc1777341cd643429649a0b5c621a99039b0e3a11f10714fd3d not found: ID does not exist"
Nov 01 15:36:11.309220 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.309203    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4ee975-5cff-493d-8107-3bc0039b01f9-kube-api-access-hx9cv\" (UniqueName: \"kubernetes.io/projected/6f4ee975-5cff-493d-8107-3bc0039b01f9-kube-api-access-hx9cv\") pod \"6f4ee975-5cff-493d-8107-3bc0039b01f9\" (UID: \"6f4ee975-5cff-493d-8107-3bc0039b01f9\") "
Nov 01 15:36:11.309328 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.309315    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4ee975-5cff-493d-8107-3bc0039b01f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4ee975-5cff-493d-8107-3bc0039b01f9-plugins-config-volume\") pod \"6f4ee975-5cff-493d-8107-3bc0039b01f9\" (UID: \"6f4ee975-5cff-493d-8107-3bc0039b01f9\") "
Nov 01 15:36:11.309647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.309627    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4ee975-5cff-493d-8107-3bc0039b01f9-plugins-config-volume" pod "6f4ee975-5cff-493d-8107-3bc0039b01f9" (UID: "6f4ee975-5cff-493d-8107-3bc0039b01f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:36:11.309744 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.309728    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgz2\" (UniqueName: \"kubernetes.io/projected/6ae0cbc5-a4e1-48ac-bc28-486970518262-kube-api-access-6zgz2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:36:11.309793 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.309746    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4ee975-5cff-493d-8107-3bc0039b01f9-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:36:11.311138 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.311119    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4ee975-5cff-493d-8107-3bc0039b01f9-kube-api-access-hx9cv" pod "6f4ee975-5cff-493d-8107-3bc0039b01f9" (UID: "6f4ee975-5cff-493d-8107-3bc0039b01f9"). InnerVolumeSpecName "kube-api-access-hx9cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:36:11.411161 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.411125    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9cv\" (UniqueName: \"kubernetes.io/projected/6f4ee975-5cff-493d-8107-3bc0039b01f9-kube-api-access-hx9cv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:36:11.483161 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.483126    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"]
Nov 01 15:36:11.485887 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.485858    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a0fa146-8b88bf59c-h8tl4"]
Nov 01 15:36:11.696228 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.696195    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"]
Nov 01 15:36:11.699997 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.699969    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a0fa146-795658b4cf-4rc4s"]
Nov 01 15:36:11.961994 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.961923    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae0cbc5-a4e1-48ac-bc28-486970518262" path="/var/lib/kubelet/pods/6ae0cbc5-a4e1-48ac-bc28-486970518262/volumes"
Nov 01 15:36:11.962365 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:36:11.962351    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4ee975-5cff-493d-8107-3bc0039b01f9" path="/var/lib/kubelet/pods/6f4ee975-5cff-493d-8107-3bc0039b01f9/volumes"
Nov 01 15:36:35.150102 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:36:35.150060    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:37:05.149272 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:37:05.149234    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:37:05.717940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:05.717910    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"]
Nov 01 15:37:05.824778 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:05.824747    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"]
Nov 01 15:37:10.719146 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:10.719110    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526" podUID="1c9692a3-6f28-4c34-b73d-297815b3845b" containerName="payload-processing" containerID="cri-o://efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83" gracePeriod=25
Nov 01 15:37:10.825515 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:10.825450    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf" podUID="342fd0eb-3d70-4109-8784-5addd7934bfb" containerName="payload-pre-processing" containerID="cri-o://f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb" gracePeriod=25
Nov 01 15:37:10.994792 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:10.994765    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:37:11.096599 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.096575    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:37:11.116094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.116067    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9692a3-6f28-4c34-b73d-297815b3845b-kube-api-access-n8hc8\" (UniqueName: \"kubernetes.io/projected/1c9692a3-6f28-4c34-b73d-297815b3845b-kube-api-access-n8hc8\") pod \"1c9692a3-6f28-4c34-b73d-297815b3845b\" (UID: \"1c9692a3-6f28-4c34-b73d-297815b3845b\") "
Nov 01 15:37:11.116245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.116118    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9692a3-6f28-4c34-b73d-297815b3845b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9692a3-6f28-4c34-b73d-297815b3845b-plugins-config-volume\") pod \"1c9692a3-6f28-4c34-b73d-297815b3845b\" (UID: \"1c9692a3-6f28-4c34-b73d-297815b3845b\") "
Nov 01 15:37:11.116550 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.116529    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9692a3-6f28-4c34-b73d-297815b3845b-plugins-config-volume" pod "1c9692a3-6f28-4c34-b73d-297815b3845b" (UID: "1c9692a3-6f28-4c34-b73d-297815b3845b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:37:11.118438 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.118414    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9692a3-6f28-4c34-b73d-297815b3845b-kube-api-access-n8hc8" pod "1c9692a3-6f28-4c34-b73d-297815b3845b" (UID: "1c9692a3-6f28-4c34-b73d-297815b3845b"). InnerVolumeSpecName "kube-api-access-n8hc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:11.217816 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.217784    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342fd0eb-3d70-4109-8784-5addd7934bfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342fd0eb-3d70-4109-8784-5addd7934bfb-plugins-config-volume\") pod \"342fd0eb-3d70-4109-8784-5addd7934bfb\" (UID: \"342fd0eb-3d70-4109-8784-5addd7934bfb\") "
Nov 01 15:37:11.217987 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.217902    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342fd0eb-3d70-4109-8784-5addd7934bfb-kube-api-access-gvx78\" (UniqueName: \"kubernetes.io/projected/342fd0eb-3d70-4109-8784-5addd7934bfb-kube-api-access-gvx78\") pod \"342fd0eb-3d70-4109-8784-5addd7934bfb\" (UID: \"342fd0eb-3d70-4109-8784-5addd7934bfb\") "
Nov 01 15:37:11.218221 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.218194    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342fd0eb-3d70-4109-8784-5addd7934bfb-plugins-config-volume" pod "342fd0eb-3d70-4109-8784-5addd7934bfb" (UID: "342fd0eb-3d70-4109-8784-5addd7934bfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:37:11.218378 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.218365    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8hc8\" (UniqueName: \"kubernetes.io/projected/1c9692a3-6f28-4c34-b73d-297815b3845b-kube-api-access-n8hc8\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:11.218463 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.218382    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9692a3-6f28-4c34-b73d-297815b3845b-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:11.218463 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.218425    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342fd0eb-3d70-4109-8784-5addd7934bfb-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:11.220086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.220065    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342fd0eb-3d70-4109-8784-5addd7934bfb-kube-api-access-gvx78" pod "342fd0eb-3d70-4109-8784-5addd7934bfb" (UID: "342fd0eb-3d70-4109-8784-5addd7934bfb"). InnerVolumeSpecName "kube-api-access-gvx78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:11.320149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.320115    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvx78\" (UniqueName: \"kubernetes.io/projected/342fd0eb-3d70-4109-8784-5addd7934bfb-kube-api-access-gvx78\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:11.694124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.694037    5423 generic.go:356] "Generic (PLEG): container finished" podID="1c9692a3-6f28-4c34-b73d-297815b3845b" containerID="efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83" exitCode=0
Nov 01 15:37:11.694124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.694095    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"
Nov 01 15:37:11.694350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.694113    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526" event={"ID":"1c9692a3-6f28-4c34-b73d-297815b3845b","Type":"ContainerDied","Data":"efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83"}
Nov 01 15:37:11.694350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.694181    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526" event={"ID":"1c9692a3-6f28-4c34-b73d-297815b3845b","Type":"ContainerDied","Data":"ec55071671e7ca6f7de87d684c8acecba0503787f84f919e8664a1713f601903"}
Nov 01 15:37:11.694350 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.694212    5423 scope.go:122] "RemoveContainer" containerID="efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83"
Nov 01 15:37:11.696057 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.696029    5423 generic.go:356] "Generic (PLEG): container finished" podID="342fd0eb-3d70-4109-8784-5addd7934bfb" containerID="f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb" exitCode=0
Nov 01 15:37:11.696186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.696076    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"
Nov 01 15:37:11.696186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.696082    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf" event={"ID":"342fd0eb-3d70-4109-8784-5addd7934bfb","Type":"ContainerDied","Data":"f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb"}
Nov 01 15:37:11.696186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.696123    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf" event={"ID":"342fd0eb-3d70-4109-8784-5addd7934bfb","Type":"ContainerDied","Data":"d7dbb063609d113eaa652274cdf96242baf84f95e7f4de816dfe985e30dc4b05"}
Nov 01 15:37:11.708518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.708497    5423 scope.go:122] "RemoveContainer" containerID="efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83"
Nov 01 15:37:11.708841 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:37:11.708806    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83\": container with ID starting with efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83 not found: ID does not exist" containerID="efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83"
Nov 01 15:37:11.708938 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.708844    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83"} err="failed to get container status \"efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83\": rpc error: code = NotFound desc = could not find container \"efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83\": container with ID starting with efd10ab94ac8673e940cfd36d62b67a6eea44feaf458e8c4fdcc71fbd84d9f83 not found: ID does not exist"
Nov 01 15:37:11.708938 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.708867    5423 scope.go:122] "RemoveContainer" containerID="f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb"
Nov 01 15:37:11.717920 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.717905    5423 scope.go:122] "RemoveContainer" containerID="f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb"
Nov 01 15:37:11.718160 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:37:11.718141    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb\": container with ID starting with f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb not found: ID does not exist" containerID="f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb"
Nov 01 15:37:11.718205 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.718167    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb"} err="failed to get container status \"f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb\": rpc error: code = NotFound desc = could not find container \"f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb\": container with ID starting with f97763e434e12add6bdbf2c5d9ce0d78c611abcd6b4257cddaccbf65542c48bb not found: ID does not exist"
Nov 01 15:37:11.861955 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.861922    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"]
Nov 01 15:37:11.865124 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.865102    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-600cc7-5fb89467f9-2q526"]
Nov 01 15:37:11.964455 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.964354    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9692a3-6f28-4c34-b73d-297815b3845b" path="/var/lib/kubelet/pods/1c9692a3-6f28-4c34-b73d-297815b3845b/volumes"
Nov 01 15:37:11.975474 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.975441    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"]
Nov 01 15:37:11.980205 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:11.980177    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-600cc7-665bfcbfd7-xtvzf"]
Nov 01 15:37:13.778578 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:13.778540    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"]
Nov 01 15:37:13.778991 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:13.778776    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" containerID="cri-o://99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24" gracePeriod=30
Nov 01 15:37:13.829178 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:13.829143    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"]
Nov 01 15:37:13.829361 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:13.829343    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" containerID="cri-o://634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e" gracePeriod=30
Nov 01 15:37:13.962003 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:13.961968    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342fd0eb-3d70-4109-8784-5addd7934bfb" path="/var/lib/kubelet/pods/342fd0eb-3d70-4109-8784-5addd7934bfb/volumes"
Nov 01 15:37:15.089208 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:15.089171    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:37:15.584180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:15.584147    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4dgqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:37:15.584341 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:15.584219    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:37:15.946101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:15.946014    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8zq9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:37:15.946381 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:15.946343    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:37:19.095071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.095041    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:37:19.145022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.144999    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:37:19.226399 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226350    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-socket\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226463    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-envoy\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226497    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-podinfo\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226556    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istiod-ca-cert\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226620    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-data\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226659    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-socket\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226691    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-kube-api-access-8zftk\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-kube-api-access-8zftk\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226694    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-socket" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226746    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-certs\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.226839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226798    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-kube-api-access-fkfnj\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-kube-api-access-fkfnj\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226850    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-certs\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226876    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istiod-ca-cert\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226931    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-token\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-token\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226969    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-podinfo\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.226971    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-data" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.227051    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-data\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.227099    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-credential-socket\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.227192    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-envoy\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.227241    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-credential-socket\") pod \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\" (UID: \"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8\") "
Nov 01 15:37:19.227922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.227303    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-token\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-token\") pod \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\" (UID: \"66cbaac4-0a96-4e67-989a-d3a216ef06d6\") "
Nov 01 15:37:19.228581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.227962    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-data" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.228581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.228007    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.228581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.228028    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.228581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.228121    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-credential-socket" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.228581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.228320    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-certs" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.228581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.228536    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-credential-socket" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.228992 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.228752    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istiod-ca-cert" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:37:19.238625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.229190    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-socket" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.238625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.229470    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istiod-ca-cert" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:37:19.238625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.229529    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-certs" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.238625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.231827    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-token" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:19.238625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.232759    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-token" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:19.238625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.234057    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-envoy" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.239820 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.239769    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-kube-api-access-fkfnj" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "kube-api-access-fkfnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:19.239820 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.239771    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-podinfo" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:37:19.239994 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.239847    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-podinfo" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:37:19.239994 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.239844    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-envoy" pod "66cbaac4-0a96-4e67-989a-d3a216ef06d6" (UID: "66cbaac4-0a96-4e67-989a-d3a216ef06d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.241171 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.241148    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-kube-api-access-8zftk" pod "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" (UID: "4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8"). InnerVolumeSpecName "kube-api-access-8zftk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:19.330501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330414    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330455    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330471    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330485    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330500    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330514    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330528    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330544    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zftk\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-kube-api-access-8zftk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330559    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330573    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkfnj\" (UniqueName: \"kubernetes.io/projected/66cbaac4-0a96-4e67-989a-d3a216ef06d6-kube-api-access-fkfnj\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330587    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330602    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330615    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330629    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330642    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.330911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.330659    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cbaac4-0a96-4e67-989a-d3a216ef06d6-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.783776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.783650    5423 generic.go:356] "Generic (PLEG): container finished" podID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerID="99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24" exitCode=0
Nov 01 15:37:19.783776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.783704    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"
Nov 01 15:37:19.783776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.783738    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" event={"ID":"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8","Type":"ContainerDied","Data":"99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24"}
Nov 01 15:37:19.784051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.783793    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l" event={"ID":"4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8","Type":"ContainerDied","Data":"1cd28fd4948a3df36c37624d40e48e3ca1293abe4d8fdea5fc09464f0134448e"}
Nov 01 15:37:19.784051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.783823    5423 scope.go:122] "RemoveContainer" containerID="99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24"
Nov 01 15:37:19.785287 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.785266    5423 generic.go:356] "Generic (PLEG): container finished" podID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerID="634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e" exitCode=0
Nov 01 15:37:19.785437 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.785320    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"
Nov 01 15:37:19.785437 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.785345    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" event={"ID":"66cbaac4-0a96-4e67-989a-d3a216ef06d6","Type":"ContainerDied","Data":"634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e"}
Nov 01 15:37:19.785437 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.785413    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf" event={"ID":"66cbaac4-0a96-4e67-989a-d3a216ef06d6","Type":"ContainerDied","Data":"6bc440573106eb27634b7238c2f60bf3487914869d3273783973b3bc2ea1edea"}
Nov 01 15:37:19.794558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.794539    5423 scope.go:122] "RemoveContainer" containerID="99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24"
Nov 01 15:37:19.794857 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:37:19.794836    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24\": container with ID starting with 99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24 not found: ID does not exist" containerID="99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24"
Nov 01 15:37:19.794964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.794862    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24"} err="failed to get container status \"99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24\": rpc error: code = NotFound desc = could not find container \"99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24\": container with ID starting with 99dc9ee5b480f6849c800e13ff418cafa5b3bea736f74f5361e80e3fd40c4a24 not found: ID does not exist"
Nov 01 15:37:19.794964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.794880    5423 scope.go:122] "RemoveContainer" containerID="634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e"
Nov 01 15:37:19.805034 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.805013    5423 scope.go:122] "RemoveContainer" containerID="634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e"
Nov 01 15:37:19.805290 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:37:19.805273    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e\": container with ID starting with 634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e not found: ID does not exist" containerID="634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e"
Nov 01 15:37:19.805342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.805297    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e"} err="failed to get container status \"634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e\": rpc error: code = NotFound desc = could not find container \"634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e\": container with ID starting with 634b2a226659e64248c79f21113f2e03cac82bc4e65532d7f2482fb35361267e not found: ID does not exist"
Nov 01 15:37:19.951675 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.951639    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"]
Nov 01 15:37:19.961093 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:19.961068    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4dgqf"]
Nov 01 15:37:20.065343 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:20.065313    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"]
Nov 01 15:37:20.073834 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:20.073807    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8zq9l"]
Nov 01 15:37:21.961857 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:21.961820    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" path="/var/lib/kubelet/pods/4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8/volumes"
Nov 01 15:37:21.962308 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:21.962292    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" path="/var/lib/kubelet/pods/66cbaac4-0a96-4e67-989a-d3a216ef06d6/volumes"
Nov 01 15:37:30.678050 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:30.678015    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"]
Nov 01 15:37:30.678480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:30.678222    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy" containerID="cri-o://6e3484d6d743af21c1f559a3ac0f6f24f7d2adb96ee318869ff614fc6047f134" gracePeriod=30
Nov 01 15:37:35.149853 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:37:35.149816    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:37:35.909784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:35.909746    5423 generic.go:356] "Generic (PLEG): container finished" podID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerID="6e3484d6d743af21c1f559a3ac0f6f24f7d2adb96ee318869ff614fc6047f134" exitCode=0
Nov 01 15:37:35.909784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:35.909777    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" event={"ID":"9f762aa0-047b-40cd-8dd5-d179e8235f44","Type":"ContainerDied","Data":"6e3484d6d743af21c1f559a3ac0f6f24f7d2adb96ee318869ff614fc6047f134"}
Nov 01 15:37:35.910033 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:35.909843    5423 scope.go:122] "RemoveContainer" containerID="c6b1157ca5c22f7ec2b6b0a20e3e934a784ea5382941ca637d9f6b02ae4e5fbd"
Nov 01 15:37:35.948709 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:35.948659    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:37:36.063887 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.063858    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-kube-api-access-8wsln\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-kube-api-access-8wsln\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064068 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.063895    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-data\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064068 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064030    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-socket\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064204 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064117    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-certs\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064204 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064144    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-data" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:36.064204 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064183    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-podinfo\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064365 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064240    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-envoy\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064365 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064278    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-socket" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:36.064365 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064352    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f762aa0-047b-40cd-8dd5-d179e8235f44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f762aa0-047b-40cd-8dd5-d179e8235f44-istiod-ca-cert\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064365 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064358    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-certs" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:36.064595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064411    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-credential-socket\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064489    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-token\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-token\") pod \"9f762aa0-047b-40cd-8dd5-d179e8235f44\" (UID: \"9f762aa0-047b-40cd-8dd5-d179e8235f44\") "
Nov 01 15:37:36.064709 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064681    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-credential-socket" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:36.064758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.064687    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f762aa0-047b-40cd-8dd5-d179e8235f44-istiod-ca-cert" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:37:36.065452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.065177    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f762aa0-047b-40cd-8dd5-d179e8235f44-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.065452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.065200    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.065452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.065215    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.065452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.065231    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.065452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.065244    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.066479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.066457    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-podinfo" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:37:36.066610 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.066587    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-envoy" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:36.066655 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.066625    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-kube-api-access-8wsln" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "kube-api-access-8wsln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:36.066751 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.066734    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-token" pod "9f762aa0-047b-40cd-8dd5-d179e8235f44" (UID: "9f762aa0-047b-40cd-8dd5-d179e8235f44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:36.166923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.166890    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.166923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.166917    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.166923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.166926    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.167761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.166936    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wsln\" (UniqueName: \"kubernetes.io/projected/9f762aa0-047b-40cd-8dd5-d179e8235f44-kube-api-access-8wsln\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:36.917599 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.917567    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"
Nov 01 15:37:36.917804 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.917600    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw" event={"ID":"9f762aa0-047b-40cd-8dd5-d179e8235f44","Type":"ContainerDied","Data":"7559c4fc88c9264cbe669cd8a226f09f687e26ef6b50982144d0dc5aa673047a"}
Nov 01 15:37:36.917804 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:36.917684    5423 scope.go:122] "RemoveContainer" containerID="6e3484d6d743af21c1f559a3ac0f6f24f7d2adb96ee318869ff614fc6047f134"
Nov 01 15:37:37.139423 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:37.139376    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"]
Nov 01 15:37:37.190860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:37.190784    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6a0fa146-openshift-default-79f5f9975c-nr8cw"]
Nov 01 15:37:37.962846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:37.962815    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" path="/var/lib/kubelet/pods/9f762aa0-047b-40cd-8dd5-d179e8235f44/volumes"
Nov 01 15:37:38.489295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:38.489258    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"]
Nov 01 15:37:38.597314 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:38.597282    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"]
Nov 01 15:37:43.149478 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:43.149442    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:37:43.177104 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:43.177084    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:37:43.489884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:43.489793    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk" podUID="84886640-64b4-4cc6-b63d-fbaee23c54bd" containerName="payload-processing" containerID="cri-o://e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec" gracePeriod=25
Nov 01 15:37:43.597779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:43.597735    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr" podUID="1e75b990-47d1-49a6-ae96-78ce34057475" containerName="payload-pre-processing" containerID="cri-o://70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9" gracePeriod=25
Nov 01 15:37:48.727306 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:48.727268    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr" podUID="1e75b990-47d1-49a6-ae96-78ce34057475" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:37:48.727763 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:48.727274    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk" podUID="84886640-64b4-4cc6-b63d-fbaee23c54bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:37:58.728271 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:58.728232    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk" podUID="84886640-64b4-4cc6-b63d-fbaee23c54bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:37:58.728794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:37:58.728236    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr" podUID="1e75b990-47d1-49a6-ae96-78ce34057475" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:38:03.754674 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:03.754610    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 15:38:03.869972 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:03.869948    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 15:38:03.909996 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:03.909968    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84886640-64b4-4cc6-b63d-fbaee23c54bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84886640-64b4-4cc6-b63d-fbaee23c54bd-plugins-config-volume\") pod \"84886640-64b4-4cc6-b63d-fbaee23c54bd\" (UID: \"84886640-64b4-4cc6-b63d-fbaee23c54bd\") "
Nov 01 15:38:03.910177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:03.910021    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84886640-64b4-4cc6-b63d-fbaee23c54bd-kube-api-access-rcwjn\" (UniqueName: \"kubernetes.io/projected/84886640-64b4-4cc6-b63d-fbaee23c54bd-kube-api-access-rcwjn\") pod \"84886640-64b4-4cc6-b63d-fbaee23c54bd\" (UID: \"84886640-64b4-4cc6-b63d-fbaee23c54bd\") "
Nov 01 15:38:03.910399 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:03.910355    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84886640-64b4-4cc6-b63d-fbaee23c54bd-plugins-config-volume" pod "84886640-64b4-4cc6-b63d-fbaee23c54bd" (UID: "84886640-64b4-4cc6-b63d-fbaee23c54bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:38:03.910596 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:03.910578    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84886640-64b4-4cc6-b63d-fbaee23c54bd-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:03.912286 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:03.912263    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84886640-64b4-4cc6-b63d-fbaee23c54bd-kube-api-access-rcwjn" pod "84886640-64b4-4cc6-b63d-fbaee23c54bd" (UID: "84886640-64b4-4cc6-b63d-fbaee23c54bd"). InnerVolumeSpecName "kube-api-access-rcwjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:38:04.011692 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.011607    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e75b990-47d1-49a6-ae96-78ce34057475-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e75b990-47d1-49a6-ae96-78ce34057475-plugins-config-volume\") pod \"1e75b990-47d1-49a6-ae96-78ce34057475\" (UID: \"1e75b990-47d1-49a6-ae96-78ce34057475\") "
Nov 01 15:38:04.011846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.011719    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e75b990-47d1-49a6-ae96-78ce34057475-kube-api-access-s7lqm\" (UniqueName: \"kubernetes.io/projected/1e75b990-47d1-49a6-ae96-78ce34057475-kube-api-access-s7lqm\") pod \"1e75b990-47d1-49a6-ae96-78ce34057475\" (UID: \"1e75b990-47d1-49a6-ae96-78ce34057475\") "
Nov 01 15:38:04.012013 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.011991    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e75b990-47d1-49a6-ae96-78ce34057475-plugins-config-volume" pod "1e75b990-47d1-49a6-ae96-78ce34057475" (UID: "1e75b990-47d1-49a6-ae96-78ce34057475"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:38:04.012285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.012257    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcwjn\" (UniqueName: \"kubernetes.io/projected/84886640-64b4-4cc6-b63d-fbaee23c54bd-kube-api-access-rcwjn\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:04.012285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.012285    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e75b990-47d1-49a6-ae96-78ce34057475-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:04.014359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.014331    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e75b990-47d1-49a6-ae96-78ce34057475-kube-api-access-s7lqm" pod "1e75b990-47d1-49a6-ae96-78ce34057475" (UID: "1e75b990-47d1-49a6-ae96-78ce34057475"). InnerVolumeSpecName "kube-api-access-s7lqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:38:04.083819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.083785    5423 generic.go:356] "Generic (PLEG): container finished" podID="84886640-64b4-4cc6-b63d-fbaee23c54bd" containerID="e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec" exitCode=0
Nov 01 15:38:04.084002 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.083833    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"
Nov 01 15:38:04.084002 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.083865    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk" event={"ID":"84886640-64b4-4cc6-b63d-fbaee23c54bd","Type":"ContainerDied","Data":"e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec"}
Nov 01 15:38:04.084002 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.083928    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk" event={"ID":"84886640-64b4-4cc6-b63d-fbaee23c54bd","Type":"ContainerDied","Data":"7e9d12540eb66e005081d063d6165c571e3370d46aaecf87d9424ab4c8d3a9cf"}
Nov 01 15:38:04.084002 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.083959    5423 scope.go:122] "RemoveContainer" containerID="e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec"
Nov 01 15:38:04.085305 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.085279    5423 generic.go:356] "Generic (PLEG): container finished" podID="1e75b990-47d1-49a6-ae96-78ce34057475" containerID="70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9" exitCode=0
Nov 01 15:38:04.085429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.085311    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr" event={"ID":"1e75b990-47d1-49a6-ae96-78ce34057475","Type":"ContainerDied","Data":"70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9"}
Nov 01 15:38:04.085429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.085345    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"
Nov 01 15:38:04.085429 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.085360    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr" event={"ID":"1e75b990-47d1-49a6-ae96-78ce34057475","Type":"ContainerDied","Data":"e7c6bd90ef0583d15641362450cfee7b579d2c9dcedbc969a92c26d041e0dfd3"}
Nov 01 15:38:04.095191 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.095173    5423 scope.go:122] "RemoveContainer" containerID="e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec"
Nov 01 15:38:04.095556 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:04.095507    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec\": container with ID starting with e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec not found: ID does not exist" containerID="e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec"
Nov 01 15:38:04.095556 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.095539    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec"} err="failed to get container status \"e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec\": rpc error: code = NotFound desc = could not find container \"e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec\": container with ID starting with e1410a2445258a7d60234aa5b1665e3656e11c9cdc99310a0d9eaee020f421ec not found: ID does not exist"
Nov 01 15:38:04.095676 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.095565    5423 scope.go:122] "RemoveContainer" containerID="70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9"
Nov 01 15:38:04.104156 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.104133    5423 scope.go:122] "RemoveContainer" containerID="70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9"
Nov 01 15:38:04.104416 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:04.104378    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9\": container with ID starting with 70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9 not found: ID does not exist" containerID="70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9"
Nov 01 15:38:04.104490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.104426    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9"} err="failed to get container status \"70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9\": rpc error: code = NotFound desc = could not find container \"70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9\": container with ID starting with 70492ae390721f2cc3b753968839a8534858aaf5b16c9d78a2964813ad6008e9 not found: ID does not exist"
Nov 01 15:38:04.113301 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.113275    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7lqm\" (UniqueName: \"kubernetes.io/projected/1e75b990-47d1-49a6-ae96-78ce34057475-kube-api-access-s7lqm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:04.250887 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.250852    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"]
Nov 01 15:38:04.257058 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.257033    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d06633-647498bdb7-bsqcr"]
Nov 01 15:38:04.368142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.368110    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"]
Nov 01 15:38:04.376783 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:04.376745    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d06633-779685445c-dglfk"]
Nov 01 15:38:05.149533 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:05.149497    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:38:05.960251 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:05.960215    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e75b990-47d1-49a6-ae96-78ce34057475" path="/var/lib/kubelet/pods/1e75b990-47d1-49a6-ae96-78ce34057475/volumes"
Nov 01 15:38:05.960658 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:05.960644    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84886640-64b4-4cc6-b63d-fbaee23c54bd" path="/var/lib/kubelet/pods/84886640-64b4-4cc6-b63d-fbaee23c54bd/volumes"
Nov 01 15:38:19.921506 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:19.921466    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:38:20.230095 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:20.230014    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"]
Nov 01 15:38:20.230367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:20.230336    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy" containerID="cri-o://14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1" gracePeriod=30
Nov 01 15:38:25.496211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.496160    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:38:25.611863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.611834    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-kube-api-access-qlnlk\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-kube-api-access-qlnlk\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612003 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.611875    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-socket\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612003 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.611919    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-envoy\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612003 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.611955    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-token\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-token\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612170 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612013    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istiod-ca-cert\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612170 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612047    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-podinfo\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612160    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-credential-socket\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612209    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-data\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612222    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-socket" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:38:25.612501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612378    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istiod-ca-cert" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:38:25.612501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612481    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-credential-socket" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:38:25.612613 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612587    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-data" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:38:25.612667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612653    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-certs\") pod \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\" (UID: \"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0\") "
Nov 01 15:38:25.612928 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.612903    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-certs" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:38:25.613148 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.613125    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:25.613250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.613178    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:25.613250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.613196    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:25.613250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.613212    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:25.613250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.613226    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:25.614235 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.614208    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-kube-api-access-qlnlk" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "kube-api-access-qlnlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:38:25.614329 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.614267    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-token" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:38:25.614329 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.614286    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-envoy" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:38:25.614329 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.614322    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-podinfo" pod "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" (UID: "c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:38:25.713990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.713954    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:25.713990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.713987    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:25.713990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.713997    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:25.714224 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:25.714006    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlnlk\" (UniqueName: \"kubernetes.io/projected/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0-kube-api-access-qlnlk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:26.234919 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.234836    5423 generic.go:356] "Generic (PLEG): container finished" podID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerID="14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1" exitCode=0
Nov 01 15:38:26.234919 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.234883    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" event={"ID":"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0","Type":"ContainerDied","Data":"14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1"}
Nov 01 15:38:26.235136 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.234928    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9" event={"ID":"c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0","Type":"ContainerDied","Data":"3a1a76accb09982c2b07e03359a3bf43bd34dc2fc38b3e7ee01a2f7a9f788842"}
Nov 01 15:38:26.235136 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.234885    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"
Nov 01 15:38:26.235136 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.234952    5423 scope.go:122] "RemoveContainer" containerID="14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1"
Nov 01 15:38:26.248706 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.248677    5423 scope.go:122] "RemoveContainer" containerID="29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4"
Nov 01 15:38:26.257729 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.257700    5423 scope.go:122] "RemoveContainer" containerID="14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1"
Nov 01 15:38:26.258007 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:26.257967    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1\": container with ID starting with 14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1 not found: ID does not exist" containerID="14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1"
Nov 01 15:38:26.258107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.258011    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1"} err="failed to get container status \"14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1\": rpc error: code = NotFound desc = could not find container \"14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1\": container with ID starting with 14d2a52714a8dd72e5928e81026e435d74bdcac4574b19706df05184817e4cb1 not found: ID does not exist"
Nov 01 15:38:26.258107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.258030    5423 scope.go:122] "RemoveContainer" containerID="29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4"
Nov 01 15:38:26.258283 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:26.258266    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4\": container with ID starting with 29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4 not found: ID does not exist" containerID="29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4"
Nov 01 15:38:26.258327 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.258291    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4"} err="failed to get container status \"29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4\": rpc error: code = NotFound desc = could not find container \"29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4\": container with ID starting with 29de98c99c7770f17ed1897b3c50c1ddeabac91a8c8d0efc04628a022af37bc4 not found: ID does not exist"
Nov 01 15:38:26.403292 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.403254    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"]
Nov 01 15:38:26.454287 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:26.454252    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-600cc7-openshift-default-fc4cd7cc-6qjd9"]
Nov 01 15:38:27.961006 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:27.960964    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" path="/var/lib/kubelet/pods/c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0/volumes"
Nov 01 15:38:35.150154 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:35.150117    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:38:50.315339 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.315259    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"]
Nov 01 15:38:50.315793 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.315497    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8" podUID="2e7b704b-d872-4d1c-bca8-d8347330da3b" containerName="maas-api" containerID="cri-o://d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b" gracePeriod=30
Nov 01 15:38:50.625299 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.625273    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 15:38:50.661486 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.661456    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls\") pod \"2e7b704b-d872-4d1c-bca8-d8347330da3b\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") "
Nov 01 15:38:50.661694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.661533    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7b704b-d872-4d1c-bca8-d8347330da3b-kube-api-access-xtqkn\" (UniqueName: \"kubernetes.io/projected/2e7b704b-d872-4d1c-bca8-d8347330da3b-kube-api-access-xtqkn\") pod \"2e7b704b-d872-4d1c-bca8-d8347330da3b\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") "
Nov 01 15:38:50.661694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.661562    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-metrics-certs\") pod \"2e7b704b-d872-4d1c-bca8-d8347330da3b\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") "
Nov 01 15:38:50.661694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.661612    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7b704b-d872-4d1c-bca8-d8347330da3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7b704b-d872-4d1c-bca8-d8347330da3b-openshift-service-ca\") pod \"2e7b704b-d872-4d1c-bca8-d8347330da3b\" (UID: \"2e7b704b-d872-4d1c-bca8-d8347330da3b\") "
Nov 01 15:38:50.662644 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.662612    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7b704b-d872-4d1c-bca8-d8347330da3b-openshift-service-ca" pod "2e7b704b-d872-4d1c-bca8-d8347330da3b" (UID: "2e7b704b-d872-4d1c-bca8-d8347330da3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:38:50.662644 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.663327    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7b704b-d872-4d1c-bca8-d8347330da3b-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:50.713591 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.685526    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-metrics-certs" pod "2e7b704b-d872-4d1c-bca8-d8347330da3b" (UID: "2e7b704b-d872-4d1c-bca8-d8347330da3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:38:50.713591 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.713439    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls" pod "2e7b704b-d872-4d1c-bca8-d8347330da3b" (UID: "2e7b704b-d872-4d1c-bca8-d8347330da3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:38:50.713591 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.713584    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7b704b-d872-4d1c-bca8-d8347330da3b-kube-api-access-xtqkn" pod "2e7b704b-d872-4d1c-bca8-d8347330da3b" (UID: "2e7b704b-d872-4d1c-bca8-d8347330da3b"). InnerVolumeSpecName "kube-api-access-xtqkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:38:50.764468 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.764438    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:50.764468 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.764468    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtqkn\" (UniqueName: \"kubernetes.io/projected/2e7b704b-d872-4d1c-bca8-d8347330da3b-kube-api-access-xtqkn\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:50.764671 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:50.764478    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7b704b-d872-4d1c-bca8-d8347330da3b-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:51.400833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.400798    5423 generic.go:356] "Generic (PLEG): container finished" podID="2e7b704b-d872-4d1c-bca8-d8347330da3b" containerID="d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b" exitCode=0
Nov 01 15:38:51.401273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.400843    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"
Nov 01 15:38:51.401273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.400859    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8" event={"ID":"2e7b704b-d872-4d1c-bca8-d8347330da3b","Type":"ContainerDied","Data":"d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b"}
Nov 01 15:38:51.401273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.400899    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8" event={"ID":"2e7b704b-d872-4d1c-bca8-d8347330da3b","Type":"ContainerDied","Data":"2a1253921fbc3326dedf64429bb49ff52a7d95da9efaf9690f4516d7c9d92c3b"}
Nov 01 15:38:51.401273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.400919    5423 scope.go:122] "RemoveContainer" containerID="d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b"
Nov 01 15:38:51.412021 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.412005    5423 scope.go:122] "RemoveContainer" containerID="d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b"
Nov 01 15:38:51.412249 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:51.412233    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b\": container with ID starting with d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b not found: ID does not exist" containerID="d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b"
Nov 01 15:38:51.412313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.412262    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b"} err="failed to get container status \"d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b\": rpc error: code = NotFound desc = could not find container \"d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b\": container with ID starting with d192c06264aec3564c22f0ac499b0ed3f8d92a984fe1fda7bbb37b5860bcc56b not found: ID does not exist"
Nov 01 15:38:51.508568 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.508535    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-a0e2b8"]
Nov 01 15:38:51.509170 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509149    5423 cpu_manager.go:414] "containerMap: removing container" podUID="84886640-64b4-4cc6-b63d-fbaee23c54bd" containerName="payload-processing"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509171    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84886640-64b4-4cc6-b63d-fbaee23c54bd" containerName="payload-processing"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509183    5423 cpu_manager.go:414] "containerMap: removing container" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509191    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509218    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6ae0cbc5-a4e1-48ac-bc28-486970518262" containerName="payload-processing"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509226    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae0cbc5-a4e1-48ac-bc28-486970518262" containerName="payload-processing"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509241    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509248    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509273    5423 cpu_manager.go:414] "containerMap: removing container" podUID="2e7b704b-d872-4d1c-bca8-d8347330da3b" containerName="maas-api"
Nov 01 15:38:51.509290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509280    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7b704b-d872-4d1c-bca8-d8347330da3b" containerName="maas-api"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509298    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6f4ee975-5cff-493d-8107-3bc0039b01f9" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509306    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4ee975-5cff-493d-8107-3bc0039b01f9" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509320    5423 cpu_manager.go:414] "containerMap: removing container" podUID="342fd0eb-3d70-4109-8784-5addd7934bfb" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509329    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342fd0eb-3d70-4109-8784-5addd7934bfb" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509347    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509355    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509367    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1c9692a3-6f28-4c34-b73d-297815b3845b" containerName="payload-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509375    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9692a3-6f28-4c34-b73d-297815b3845b" containerName="payload-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509402    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509410    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509423    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1e75b990-47d1-49a6-ae96-78ce34057475" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509432    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e75b990-47d1-49a6-ae96-78ce34057475" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509459    5423 cpu_manager.go:414] "containerMap: removing container" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509466    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509610    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7b704b-d872-4d1c-bca8-d8347330da3b" containerName="maas-api"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509625    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4ee975-5cff-493d-8107-3bc0039b01f9" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509637    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="342fd0eb-3d70-4109-8784-5addd7934bfb" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509649    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ffdd6ed-54e1-4ec2-8773-dc8f9d46edc8" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509659    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9692a3-6f28-4c34-b73d-297815b3845b" containerName="payload-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509669    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509678    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509687    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e75b990-47d1-49a6-ae96-78ce34057475" containerName="payload-pre-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509698    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="66cbaac4-0a96-4e67-989a-d3a216ef06d6" containerName="istio-proxy"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509711    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="84886640-64b4-4cc6-b63d-fbaee23c54bd" containerName="payload-processing"
Nov 01 15:38:51.509812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.509724    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae0cbc5-a4e1-48ac-bc28-486970518262" containerName="payload-processing"
Nov 01 15:38:51.513359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.513340    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-a0e2b8"
Nov 01 15:38:51.564818 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.564787    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-a0e2b8"]
Nov 01 15:38:51.567006 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.566976    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jngw\""
Nov 01 15:38:51.569733 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.569712    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"]
Nov 01 15:38:51.621807 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.621764    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d06633-56b7d8b4f8-sm4b8"]
Nov 01 15:38:51.677244 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.677162    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvg6d\" (UniqueName: \"kubernetes.io/projected/fcc38969-b846-457d-ae8c-2c4280e59a56-kube-api-access-xvg6d\") pod \"test-curl-7384-a0e2b8\" (UID: \"fcc38969-b846-457d-ae8c-2c4280e59a56\") " pod="opendatahub/test-curl-7384-a0e2b8"
Nov 01 15:38:51.778139 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.778108    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvg6d\" (UniqueName: \"kubernetes.io/projected/fcc38969-b846-457d-ae8c-2c4280e59a56-kube-api-access-xvg6d\") pod \"test-curl-7384-a0e2b8\" (UID: \"fcc38969-b846-457d-ae8c-2c4280e59a56\") " pod="opendatahub/test-curl-7384-a0e2b8"
Nov 01 15:38:51.839775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.839750    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvg6d\" (UniqueName: \"kubernetes.io/projected/fcc38969-b846-457d-ae8c-2c4280e59a56-kube-api-access-xvg6d\") pod \"test-curl-7384-a0e2b8\" (UID: \"fcc38969-b846-457d-ae8c-2c4280e59a56\") " pod="opendatahub/test-curl-7384-a0e2b8"
Nov 01 15:38:51.960784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:51.960709    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7b704b-d872-4d1c-bca8-d8347330da3b" path="/var/lib/kubelet/pods/2e7b704b-d872-4d1c-bca8-d8347330da3b/volumes"
Nov 01 15:38:52.134446 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:52.128180    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-a0e2b8"
Nov 01 15:38:52.514853 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:38:52.514812    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcc38969_b846_457d_ae8c_2c4280e59a56.slice/crio-f70e2423cc3341f54338f91a6663961c502346956f7066e3427c8d49b42458e4 WatchSource:0}: Error finding container f70e2423cc3341f54338f91a6663961c502346956f7066e3427c8d49b42458e4: Status 404 returned error can't find the container with id f70e2423cc3341f54338f91a6663961c502346956f7066e3427c8d49b42458e4
Nov 01 15:38:52.517800 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:52.517743    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-a0e2b8"]
Nov 01 15:38:53.417249 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:53.417209    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-a0e2b8" event={"ID":"fcc38969-b846-457d-ae8c-2c4280e59a56","Type":"ContainerStarted","Data":"3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb"}
Nov 01 15:38:53.417249 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:53.417255    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-a0e2b8" event={"ID":"fcc38969-b846-457d-ae8c-2c4280e59a56","Type":"ContainerStarted","Data":"f70e2423cc3341f54338f91a6663961c502346956f7066e3427c8d49b42458e4"}
Nov 01 15:38:53.532284 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:53.532231    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-a0e2b8" podStartSLOduration=2.5322181009999998 podStartE2EDuration="2.532218101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:38:53.530130092 +0000 UTC m=+4873.695615390" watchObservedRunningTime="2026-11-01 15:38:53.532218101 +0000 UTC m=+4873.697703399"
Nov 01 15:38:54.979572 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:54.979536    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-a0e2b8"]
Nov 01 15:38:54.984270 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:54.984239    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-a0e2b8"]
Nov 01 15:38:55.118073 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.118038    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-542e7e"]
Nov 01 15:38:55.118662 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.118648    5423 cpu_manager.go:414] "containerMap: removing container" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy"
Nov 01 15:38:55.118721 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.118663    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy"
Nov 01 15:38:55.118721 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.118678    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fcc38969-b846-457d-ae8c-2c4280e59a56" containerName="test-curl-7384-a0e2b8"
Nov 01 15:38:55.118721 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.118684    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc38969-b846-457d-ae8c-2c4280e59a56" containerName="test-curl-7384-a0e2b8"
Nov 01 15:38:55.118831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.118761    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc38969-b846-457d-ae8c-2c4280e59a56" containerName="test-curl-7384-a0e2b8"
Nov 01 15:38:55.118831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.118774    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67c9c20-ffd7-4b60-89ce-2a6c7868a5e0" containerName="istio-proxy"
Nov 01 15:38:55.118831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.118782    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f762aa0-047b-40cd-8dd5-d179e8235f44" containerName="istio-proxy"
Nov 01 15:38:55.125441 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.125420    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-542e7e"
Nov 01 15:38:55.187297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.187254    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-542e7e"]
Nov 01 15:38:55.221107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.221068    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b796w\" (UniqueName: \"kubernetes.io/projected/31998e3a-cad5-4d68-8514-0daa9a09f09b-kube-api-access-b796w\") pod \"test-curl-7384-542e7e\" (UID: \"31998e3a-cad5-4d68-8514-0daa9a09f09b\") " pod="opendatahub/test-curl-7384-542e7e"
Nov 01 15:38:55.322193 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.322155    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b796w\" (UniqueName: \"kubernetes.io/projected/31998e3a-cad5-4d68-8514-0daa9a09f09b-kube-api-access-b796w\") pod \"test-curl-7384-542e7e\" (UID: \"31998e3a-cad5-4d68-8514-0daa9a09f09b\") " pod="opendatahub/test-curl-7384-542e7e"
Nov 01 15:38:55.386023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.385989    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b796w\" (UniqueName: \"kubernetes.io/projected/31998e3a-cad5-4d68-8514-0daa9a09f09b-kube-api-access-b796w\") pod \"test-curl-7384-542e7e\" (UID: \"31998e3a-cad5-4d68-8514-0daa9a09f09b\") " pod="opendatahub/test-curl-7384-542e7e"
Nov 01 15:38:55.429115 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.429061    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-a0e2b8" podUID="fcc38969-b846-457d-ae8c-2c4280e59a56" containerName="test-curl-7384-a0e2b8" containerID="cri-o://3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb" gracePeriod=2
Nov 01 15:38:55.437644 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.437620    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-542e7e"
Nov 01 15:38:55.482100 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:55.482046    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-a0e2b8\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="fcc38969-b846-457d-ae8c-2c4280e59a56" pod="opendatahub/test-curl-7384-a0e2b8"
Nov 01 15:38:55.811367 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:38:55.811334    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31998e3a_cad5_4d68_8514_0daa9a09f09b.slice/crio-3eddfd45bc95c5b2fea08d439a2736ac5d5c736dd06c51a0218cfac32b292401 WatchSource:0}: Error finding container 3eddfd45bc95c5b2fea08d439a2736ac5d5c736dd06c51a0218cfac32b292401: Status 404 returned error can't find the container with id 3eddfd45bc95c5b2fea08d439a2736ac5d5c736dd06c51a0218cfac32b292401
Nov 01 15:38:55.814325 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:55.814299    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-542e7e"]
Nov 01 15:38:56.436097 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:56.436057    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-542e7e" event={"ID":"31998e3a-cad5-4d68-8514-0daa9a09f09b","Type":"ContainerStarted","Data":"e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786"}
Nov 01 15:38:56.436097 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:56.436101    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-542e7e" event={"ID":"31998e3a-cad5-4d68-8514-0daa9a09f09b","Type":"ContainerStarted","Data":"3eddfd45bc95c5b2fea08d439a2736ac5d5c736dd06c51a0218cfac32b292401"}
Nov 01 15:38:56.549815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:56.549759    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-542e7e" podStartSLOduration=1.5497450160000001 podStartE2EDuration="1.549745016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:38:56.549203248 +0000 UTC m=+4876.714688546" watchObservedRunningTime="2026-11-01 15:38:56.549745016 +0000 UTC m=+4876.715230311"
Nov 01 15:38:57.680859 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:57.680833    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-a0e2b8"
Nov 01 15:38:57.757936 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:57.757866    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc38969-b846-457d-ae8c-2c4280e59a56-kube-api-access-xvg6d\" (UniqueName: \"kubernetes.io/projected/fcc38969-b846-457d-ae8c-2c4280e59a56-kube-api-access-xvg6d\") pod \"fcc38969-b846-457d-ae8c-2c4280e59a56\" (UID: \"fcc38969-b846-457d-ae8c-2c4280e59a56\") "
Nov 01 15:38:57.759891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:57.759868    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc38969-b846-457d-ae8c-2c4280e59a56-kube-api-access-xvg6d" pod "fcc38969-b846-457d-ae8c-2c4280e59a56" (UID: "fcc38969-b846-457d-ae8c-2c4280e59a56"). InnerVolumeSpecName "kube-api-access-xvg6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:38:57.859722 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:57.859691    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvg6d\" (UniqueName: \"kubernetes.io/projected/fcc38969-b846-457d-ae8c-2c4280e59a56-kube-api-access-xvg6d\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:57.929242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:57.929168    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-542e7e"]
Nov 01 15:38:57.931482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:57.931456    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-542e7e"]
Nov 01 15:38:57.962511 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:57.962482    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc38969-b846-457d-ae8c-2c4280e59a56" path="/var/lib/kubelet/pods/fcc38969-b846-457d-ae8c-2c4280e59a56/volumes"
Nov 01 15:38:58.450430 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:58.450374    5423 generic.go:356] "Generic (PLEG): container finished" podID="fcc38969-b846-457d-ae8c-2c4280e59a56" containerID="3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb" exitCode=137
Nov 01 15:38:58.450633 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:58.450438    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-a0e2b8"
Nov 01 15:38:58.450633 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:58.450515    5423 scope.go:122] "RemoveContainer" containerID="3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb"
Nov 01 15:38:58.450752 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:58.450646    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-542e7e" podUID="31998e3a-cad5-4d68-8514-0daa9a09f09b" containerName="test-curl-7384-542e7e" containerID="cri-o://e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786" gracePeriod=2
Nov 01 15:38:58.463167 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:58.463137    5423 scope.go:122] "RemoveContainer" containerID="3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb"
Nov 01 15:38:58.463468 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:38:58.463442    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb\": container with ID starting with 3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb not found: ID does not exist" containerID="3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb"
Nov 01 15:38:58.463536 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:38:58.463480    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb"} err="failed to get container status \"3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb\": rpc error: code = NotFound desc = could not find container \"3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb\": container with ID starting with 3534b8366ce7d64d66c39992c3ac8c1980414e938796ada23fb2ea5e173d1ceb not found: ID does not exist"
Nov 01 15:39:00.715130 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:00.715072    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-542e7e"
Nov 01 15:39:00.794601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:00.794564    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31998e3a-cad5-4d68-8514-0daa9a09f09b-kube-api-access-b796w\" (UniqueName: \"kubernetes.io/projected/31998e3a-cad5-4d68-8514-0daa9a09f09b-kube-api-access-b796w\") pod \"31998e3a-cad5-4d68-8514-0daa9a09f09b\" (UID: \"31998e3a-cad5-4d68-8514-0daa9a09f09b\") "
Nov 01 15:39:00.796853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:00.796820    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31998e3a-cad5-4d68-8514-0daa9a09f09b-kube-api-access-b796w" pod "31998e3a-cad5-4d68-8514-0daa9a09f09b" (UID: "31998e3a-cad5-4d68-8514-0daa9a09f09b"). InnerVolumeSpecName "kube-api-access-b796w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:39:00.896047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:00.895995    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b796w\" (UniqueName: \"kubernetes.io/projected/31998e3a-cad5-4d68-8514-0daa9a09f09b-kube-api-access-b796w\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:01.471956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:01.471922    5423 generic.go:356] "Generic (PLEG): container finished" podID="31998e3a-cad5-4d68-8514-0daa9a09f09b" containerID="e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786" exitCode=137
Nov 01 15:39:01.472102 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:01.471966    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-542e7e"
Nov 01 15:39:01.472102 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:01.472010    5423 scope.go:122] "RemoveContainer" containerID="e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786"
Nov 01 15:39:01.482533 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:01.482517    5423 scope.go:122] "RemoveContainer" containerID="e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786"
Nov 01 15:39:01.482789 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:39:01.482770    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786\": container with ID starting with e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786 not found: ID does not exist" containerID="e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786"
Nov 01 15:39:01.482838 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:01.482798    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786"} err="failed to get container status \"e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786\": rpc error: code = NotFound desc = could not find container \"e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786\": container with ID starting with e9dcde4db07c1301c4707ad7c6a2321f5b91476a323a551058cf7513cbc2e786 not found: ID does not exist"
Nov 01 15:39:01.534205 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:39:01.534155    5423 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-542e7e\" is forbidden: User \"system:node:ip-10-0-135-142.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-142.us-west-2.compute.internal' and this object" podUID="31998e3a-cad5-4d68-8514-0daa9a09f09b" pod="opendatahub/test-curl-7384-542e7e"
Nov 01 15:39:01.961211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:01.961179    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31998e3a-cad5-4d68-8514-0daa9a09f09b" path="/var/lib/kubelet/pods/31998e3a-cad5-4d68-8514-0daa9a09f09b/volumes"
Nov 01 15:39:05.149560 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:39:05.149523    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:39:09.333990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:09.333956    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"]
Nov 01 15:39:14.334889 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:14.334807    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m" podUID="9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" containerName="payload-pre-processing" containerID="cri-o://579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597" gracePeriod=25
Nov 01 15:39:21.020579 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:21.020540    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m" podUID="9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:39:25.600108 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:25.600071    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:39:31.020220 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:31.020175    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m" podUID="9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:39:34.613447 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.613422    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:39:34.676083 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.676052    5423 generic.go:356] "Generic (PLEG): container finished" podID="9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" containerID="579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597" exitCode=0
Nov 01 15:39:34.676458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.676104    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"
Nov 01 15:39:34.676458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.676142    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m" event={"ID":"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3","Type":"ContainerDied","Data":"579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597"}
Nov 01 15:39:34.676458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.676199    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m" event={"ID":"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3","Type":"ContainerDied","Data":"fd716e0cfe1154626f7083ab387182e30374e549f715a3dc285dcaf967bd70e6"}
Nov 01 15:39:34.676458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.676212    5423 scope.go:122] "RemoveContainer" containerID="579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597"
Nov 01 15:39:34.685732 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.685714    5423 scope.go:122] "RemoveContainer" containerID="579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597"
Nov 01 15:39:34.686006 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:39:34.685986    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597\": container with ID starting with 579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597 not found: ID does not exist" containerID="579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597"
Nov 01 15:39:34.686090 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.686015    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597"} err="failed to get container status \"579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597\": rpc error: code = NotFound desc = could not find container \"579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597\": container with ID starting with 579cd950600a9d5e44e2b211c98b3a53197f4c0efb442ab24e0bcdc63709c597 not found: ID does not exist"
Nov 01 15:39:34.718178 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.718154    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-kube-api-access-5mw95\" (UniqueName: \"kubernetes.io/projected/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-kube-api-access-5mw95\") pod \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\" (UID: \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\") "
Nov 01 15:39:34.718310 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.718196    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-plugins-config-volume\") pod \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\" (UID: \"9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3\") "
Nov 01 15:39:34.718626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.718605    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-plugins-config-volume" pod "9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" (UID: "9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:39:34.718746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.718726    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:34.720248 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.720230    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-kube-api-access-5mw95" pod "9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" (UID: "9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3"). InnerVolumeSpecName "kube-api-access-5mw95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:39:34.820072 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:34.820046    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mw95\" (UniqueName: \"kubernetes.io/projected/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3-kube-api-access-5mw95\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:35.098614 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:35.098580    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"]
Nov 01 15:39:35.149915 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:35.149883    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b3f06c-57f55f5b7-jwq8m"]
Nov 01 15:39:35.150081 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:39:35.149941    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:39:35.960270 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:35.960238    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" path="/var/lib/kubelet/pods/9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3/volumes"
Nov 01 15:39:45.170884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.170773    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"]
Nov 01 15:39:45.177760 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.170979    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n" podUID="bd1dec20-0f8a-4623-8f6f-683fd454ede4" containerName="maas-api" containerID="cri-o://cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc" gracePeriod=30
Nov 01 15:39:45.458146 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.458120    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:39:45.550988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.550948    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1dec20-0f8a-4623-8f6f-683fd454ede4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1dec20-0f8a-4623-8f6f-683fd454ede4-openshift-service-ca\") pod \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") "
Nov 01 15:39:45.551913 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.551890    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-metrics-certs\") pod \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") "
Nov 01 15:39:45.552577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.552545    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls\") pod \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") "
Nov 01 15:39:45.566024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.551761    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1dec20-0f8a-4623-8f6f-683fd454ede4-openshift-service-ca" pod "bd1dec20-0f8a-4623-8f6f-683fd454ede4" (UID: "bd1dec20-0f8a-4623-8f6f-683fd454ede4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:39:45.566024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.560763    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-metrics-certs" pod "bd1dec20-0f8a-4623-8f6f-683fd454ede4" (UID: "bd1dec20-0f8a-4623-8f6f-683fd454ede4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:39:45.566024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.563524    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1dec20-0f8a-4623-8f6f-683fd454ede4-kube-api-access-qwrjm\" (UniqueName: \"kubernetes.io/projected/bd1dec20-0f8a-4623-8f6f-683fd454ede4-kube-api-access-qwrjm\") pod \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\" (UID: \"bd1dec20-0f8a-4623-8f6f-683fd454ede4\") "
Nov 01 15:39:45.566024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.564179    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:45.566024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.564203    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1dec20-0f8a-4623-8f6f-683fd454ede4-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:45.566487 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.566044    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls" pod "bd1dec20-0f8a-4623-8f6f-683fd454ede4" (UID: "bd1dec20-0f8a-4623-8f6f-683fd454ede4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:39:45.566824 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.566800    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1dec20-0f8a-4623-8f6f-683fd454ede4-kube-api-access-qwrjm" pod "bd1dec20-0f8a-4623-8f6f-683fd454ede4" (UID: "bd1dec20-0f8a-4623-8f6f-683fd454ede4"). InnerVolumeSpecName "kube-api-access-qwrjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:39:45.665204 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.665167    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd1dec20-0f8a-4623-8f6f-683fd454ede4-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:45.665204 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.665202    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrjm\" (UniqueName: \"kubernetes.io/projected/bd1dec20-0f8a-4623-8f6f-683fd454ede4-kube-api-access-qwrjm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:45.744480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.744402    5423 generic.go:356] "Generic (PLEG): container finished" podID="bd1dec20-0f8a-4623-8f6f-683fd454ede4" containerID="cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc" exitCode=0
Nov 01 15:39:45.744480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.744444    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n" event={"ID":"bd1dec20-0f8a-4623-8f6f-683fd454ede4","Type":"ContainerDied","Data":"cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc"}
Nov 01 15:39:45.744694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.744489    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n" event={"ID":"bd1dec20-0f8a-4623-8f6f-683fd454ede4","Type":"ContainerDied","Data":"6c024f641007b5d28fc4ddf8f7fa56f1fbd8fbe345c7c99497e5e242ac9607a4"}
Nov 01 15:39:45.744694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.744492    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"
Nov 01 15:39:45.744694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.744514    5423 scope.go:122] "RemoveContainer" containerID="cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc"
Nov 01 15:39:45.754373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.754354    5423 scope.go:122] "RemoveContainer" containerID="cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc"
Nov 01 15:39:45.754674 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:39:45.754652    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc\": container with ID starting with cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc not found: ID does not exist" containerID="cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc"
Nov 01 15:39:45.754725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.754682    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc"} err="failed to get container status \"cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc\": rpc error: code = NotFound desc = could not find container \"cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc\": container with ID starting with cc7bae012945e5be2f84d6fc7e83c023ca083a5ed278a2fdbe78a7dfb97748dc not found: ID does not exist"
Nov 01 15:39:45.911645 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.911612    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"]
Nov 01 15:39:45.915007 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.914979    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-974865-6db4949788-mxr2n"]
Nov 01 15:39:45.961728 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:39:45.961695    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1dec20-0f8a-4623-8f6f-683fd454ede4" path="/var/lib/kubelet/pods/bd1dec20-0f8a-4623-8f6f-683fd454ede4/volumes"
Nov 01 15:40:05.171898 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:40:05.171858    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:40:09.560902 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:09.560866    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"]
Nov 01 15:40:24.590797 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.590696    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw" podUID="05eced9e-3508-482d-9f58-90f9e0e16546" containerName="main" containerID="cri-o://47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987" gracePeriod=45
Nov 01 15:40:24.896891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.896865    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:40:24.979280 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.979242    5423 generic.go:356] "Generic (PLEG): container finished" podID="05eced9e-3508-482d-9f58-90f9e0e16546" containerID="47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987" exitCode=0
Nov 01 15:40:24.979481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.979300    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"
Nov 01 15:40:24.979481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.979327    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw" event={"ID":"05eced9e-3508-482d-9f58-90f9e0e16546","Type":"ContainerDied","Data":"47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987"}
Nov 01 15:40:24.979481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.979375    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw" event={"ID":"05eced9e-3508-482d-9f58-90f9e0e16546","Type":"ContainerDied","Data":"0b65806c1ec4528e03189dc21402f6ed92810f3a516eb382b8728debb68ebe7c"}
Nov 01 15:40:24.979481 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.979408    5423 scope.go:122] "RemoveContainer" containerID="47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987"
Nov 01 15:40:24.989695 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.989676    5423 scope.go:122] "RemoveContainer" containerID="47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987"
Nov 01 15:40:24.989947 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:40:24.989930    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987\": container with ID starting with 47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987 not found: ID does not exist" containerID="47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987"
Nov 01 15:40:24.989995 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:24.989958    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987"} err="failed to get container status \"47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987\": rpc error: code = NotFound desc = could not find container \"47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987\": container with ID starting with 47d19b467e804689b5927e974dcc47afbcdff2aa6370ba2846cfe9a3a93cc987 not found: ID does not exist"
Nov 01 15:40:25.047140 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047114    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-home\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-home\") pod \"05eced9e-3508-482d-9f58-90f9e0e16546\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") "
Nov 01 15:40:25.047300 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047195    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05eced9e-3508-482d-9f58-90f9e0e16546-kube-api-access-5vphx\" (UniqueName: \"kubernetes.io/projected/05eced9e-3508-482d-9f58-90f9e0e16546-kube-api-access-5vphx\") pod \"05eced9e-3508-482d-9f58-90f9e0e16546\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") "
Nov 01 15:40:25.047300 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047260    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05eced9e-3508-482d-9f58-90f9e0e16546-tls-certs\" (UniqueName: \"kubernetes.io/secret/05eced9e-3508-482d-9f58-90f9e0e16546-tls-certs\") pod \"05eced9e-3508-482d-9f58-90f9e0e16546\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") "
Nov 01 15:40:25.047459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047339    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-tmp-dir\") pod \"05eced9e-3508-482d-9f58-90f9e0e16546\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") "
Nov 01 15:40:25.047459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047438    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-dshm\") pod \"05eced9e-3508-482d-9f58-90f9e0e16546\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") "
Nov 01 15:40:25.047570 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047468    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-home" pod "05eced9e-3508-482d-9f58-90f9e0e16546" (UID: "05eced9e-3508-482d-9f58-90f9e0e16546"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:25.047570 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047500    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-model-cache\") pod \"05eced9e-3508-482d-9f58-90f9e0e16546\" (UID: \"05eced9e-3508-482d-9f58-90f9e0e16546\") "
Nov 01 15:40:25.047672 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047644    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-tmp-dir" pod "05eced9e-3508-482d-9f58-90f9e0e16546" (UID: "05eced9e-3508-482d-9f58-90f9e0e16546"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:25.047934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047870    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-model-cache" pod "05eced9e-3508-482d-9f58-90f9e0e16546" (UID: "05eced9e-3508-482d-9f58-90f9e0e16546"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:25.048045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.047998    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.048045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.048021    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.048045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.048032    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.049918 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.049889    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05eced9e-3508-482d-9f58-90f9e0e16546-tls-certs" pod "05eced9e-3508-482d-9f58-90f9e0e16546" (UID: "05eced9e-3508-482d-9f58-90f9e0e16546"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:25.050248 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.050228    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-dshm" pod "05eced9e-3508-482d-9f58-90f9e0e16546" (UID: "05eced9e-3508-482d-9f58-90f9e0e16546"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:25.050517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.050492    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eced9e-3508-482d-9f58-90f9e0e16546-kube-api-access-5vphx" pod "05eced9e-3508-482d-9f58-90f9e0e16546" (UID: "05eced9e-3508-482d-9f58-90f9e0e16546"). InnerVolumeSpecName "kube-api-access-5vphx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:25.163465 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.158919    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vphx\" (UniqueName: \"kubernetes.io/projected/05eced9e-3508-482d-9f58-90f9e0e16546-kube-api-access-5vphx\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.163465 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.158957    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05eced9e-3508-482d-9f58-90f9e0e16546-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.163465 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.158971    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eced9e-3508-482d-9f58-90f9e0e16546-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.323290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.323261    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"]
Nov 01 15:40:25.323511 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.323490    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g" podUID="354e828e-9394-4e85-894b-f2930448c67d" containerName="maas-api" containerID="cri-o://870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b" gracePeriod=30
Nov 01 15:40:25.403289 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.403254    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"]
Nov 01 15:40:25.406531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.406506    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw"]
Nov 01 15:40:25.627121 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.627097    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 15:40:25.666528 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.666504    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354e828e-9394-4e85-894b-f2930448c67d-kube-api-access-lrrgl\" (UniqueName: \"kubernetes.io/projected/354e828e-9394-4e85-894b-f2930448c67d-kube-api-access-lrrgl\") pod \"354e828e-9394-4e85-894b-f2930448c67d\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") "
Nov 01 15:40:25.666616 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.666600    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-maas-api-tls\") pod \"354e828e-9394-4e85-894b-f2930448c67d\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") "
Nov 01 15:40:25.666667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.666651    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-metrics-certs\") pod \"354e828e-9394-4e85-894b-f2930448c67d\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") "
Nov 01 15:40:25.667047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.666809    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354e828e-9394-4e85-894b-f2930448c67d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354e828e-9394-4e85-894b-f2930448c67d-openshift-service-ca\") pod \"354e828e-9394-4e85-894b-f2930448c67d\" (UID: \"354e828e-9394-4e85-894b-f2930448c67d\") "
Nov 01 15:40:25.667400 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.667364    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354e828e-9394-4e85-894b-f2930448c67d-openshift-service-ca" pod "354e828e-9394-4e85-894b-f2930448c67d" (UID: "354e828e-9394-4e85-894b-f2930448c67d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:40:25.668871 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.668849    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-maas-api-tls" pod "354e828e-9394-4e85-894b-f2930448c67d" (UID: "354e828e-9394-4e85-894b-f2930448c67d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:25.669169 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.669143    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-metrics-certs" pod "354e828e-9394-4e85-894b-f2930448c67d" (UID: "354e828e-9394-4e85-894b-f2930448c67d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:25.669259 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.669162    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354e828e-9394-4e85-894b-f2930448c67d-kube-api-access-lrrgl" pod "354e828e-9394-4e85-894b-f2930448c67d" (UID: "354e828e-9394-4e85-894b-f2930448c67d"). InnerVolumeSpecName "kube-api-access-lrrgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:25.768638 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.768564    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354e828e-9394-4e85-894b-f2930448c67d-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.768638 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.768592    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrgl\" (UniqueName: \"kubernetes.io/projected/354e828e-9394-4e85-894b-f2930448c67d-kube-api-access-lrrgl\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.768638 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.768603    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.768638 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.768622    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354e828e-9394-4e85-894b-f2930448c67d-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.834451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.834377    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ca0f9e/sub-test-model-ca0f9e-kserve-546858d969-m2hnw" podUID="05eced9e-3508-482d-9f58-90f9e0e16546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.238:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:40:25.960332 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.960295    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05eced9e-3508-482d-9f58-90f9e0e16546" path="/var/lib/kubelet/pods/05eced9e-3508-482d-9f58-90f9e0e16546/volumes"
Nov 01 15:40:25.987745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.987715    5423 generic.go:356] "Generic (PLEG): container finished" podID="354e828e-9394-4e85-894b-f2930448c67d" containerID="870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b" exitCode=0
Nov 01 15:40:25.987898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.987765    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"
Nov 01 15:40:25.987898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.987810    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g" event={"ID":"354e828e-9394-4e85-894b-f2930448c67d","Type":"ContainerDied","Data":"870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b"}
Nov 01 15:40:25.987898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.987861    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g" event={"ID":"354e828e-9394-4e85-894b-f2930448c67d","Type":"ContainerDied","Data":"3a6d4df6596eae0e0c0243b10276d0ac3cf1364417868e91733de84f63772dbe"}
Nov 01 15:40:25.987898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.987886    5423 scope.go:122] "RemoveContainer" containerID="870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b"
Nov 01 15:40:25.997681 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.997662    5423 scope.go:122] "RemoveContainer" containerID="870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b"
Nov 01 15:40:25.997928 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:40:25.997909    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b\": container with ID starting with 870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b not found: ID does not exist" containerID="870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b"
Nov 01 15:40:25.997983 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:25.997939    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b"} err="failed to get container status \"870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b\": rpc error: code = NotFound desc = could not find container \"870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b\": container with ID starting with 870d950f1642e57bf55bf3f37a556f72731135a2a5a4a051b0bdfadd7eeff93b not found: ID does not exist"
Nov 01 15:40:26.154275 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:26.154246    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"]
Nov 01 15:40:26.159817 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:26.159792    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4e07cb-64cdd5dc69-9zc9g"]
Nov 01 15:40:27.960342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:27.960312    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354e828e-9394-4e85-894b-f2930448c67d" path="/var/lib/kubelet/pods/354e828e-9394-4e85-894b-f2930448c67d/volumes"
Nov 01 15:40:29.264807 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:29.264771    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"]
Nov 01 15:40:29.265340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:29.265049    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" containerID="cri-o://e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c" gracePeriod=30
Nov 01 15:40:29.882247 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:29.882209    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"]
Nov 01 15:40:33.079412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:33.079344    5423 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:40:33.079835 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:33.079454    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:40:34.530533 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.530475    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 15:40:34.703670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.703617    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-podinfo\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.703670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.703671    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-envoy\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.703944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.703713    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-data\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.703944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.703755    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-token\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-token\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.703944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.703833    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istiod-ca-cert\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.703944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.703899    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-certs\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.704184 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.703958    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-credential-socket\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.704184 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704018    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-data" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:34.704184 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704035    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-kube-api-access-7l9pz\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-kube-api-access-7l9pz\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.704361 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704099    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-socket\") pod \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\" (UID: \"bbb5f2ae-d498-4cde-95be-e7dd1339c347\") "
Nov 01 15:40:34.704462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704321    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-credential-socket" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:34.704462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704238    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-certs" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:34.704462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704358    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istiod-ca-cert" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:40:34.704980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704518    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-socket" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:34.704980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704778    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:34.704980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704794    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:34.704980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704803    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:34.704980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704812    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:34.704980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.704820    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:34.706024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.705990    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-podinfo" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:40:34.706235 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.706216    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-envoy" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:34.706532 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.706512    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-token" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:34.706532 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.706521    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-kube-api-access-7l9pz" pod "bbb5f2ae-d498-4cde-95be-e7dd1339c347" (UID: "bbb5f2ae-d498-4cde-95be-e7dd1339c347"). InnerVolumeSpecName "kube-api-access-7l9pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:34.805771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.805696    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l9pz\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-kube-api-access-7l9pz\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:34.805771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.805724    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:34.805771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.805734    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:34.805771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:34.805742    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb5f2ae-d498-4cde-95be-e7dd1339c347-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:35.043868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.043827    5423 generic.go:356] "Generic (PLEG): container finished" podID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerID="e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c" exitCode=0
Nov 01 15:40:35.044061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.043882    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"
Nov 01 15:40:35.044061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.043910    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" event={"ID":"bbb5f2ae-d498-4cde-95be-e7dd1339c347","Type":"ContainerDied","Data":"e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c"}
Nov 01 15:40:35.044061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.043964    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v" event={"ID":"bbb5f2ae-d498-4cde-95be-e7dd1339c347","Type":"ContainerDied","Data":"fcd86eacf56071ace5a8799a0b6e90025a7a3bbafd0d57b7d6da631c21647ce4"}
Nov 01 15:40:35.044061 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.043989    5423 scope.go:122] "RemoveContainer" containerID="e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c"
Nov 01 15:40:35.054666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.054648    5423 scope.go:122] "RemoveContainer" containerID="609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32"
Nov 01 15:40:35.063132 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.063116    5423 scope.go:122] "RemoveContainer" containerID="e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c"
Nov 01 15:40:35.063433 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:40:35.063414    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c\": container with ID starting with e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c not found: ID does not exist" containerID="e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c"
Nov 01 15:40:35.063498 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.063441    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c"} err="failed to get container status \"e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c\": rpc error: code = NotFound desc = could not find container \"e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c\": container with ID starting with e0e4b22373021daca4d7ff68f7859fe64afdeb0a56bdcfcf66dceebd4085ef5c not found: ID does not exist"
Nov 01 15:40:35.063498 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.063457    5423 scope.go:122] "RemoveContainer" containerID="609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32"
Nov 01 15:40:35.063694 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:40:35.063679    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32\": container with ID starting with 609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32 not found: ID does not exist" containerID="609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32"
Nov 01 15:40:35.063736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.063698    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32"} err="failed to get container status \"609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32\": rpc error: code = NotFound desc = could not find container \"609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32\": container with ID starting with 609a49ebfe91bb4d26723262cc4d4f388f755abd3fba8a312b0f12a368a9eb32 not found: ID does not exist"
Nov 01 15:40:35.149182 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:40:35.149149    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:40:35.212489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.212455    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"]
Nov 01 15:40:35.262560 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.262528    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4e07cb-openshift-default-666bf8b989-slp8v"]
Nov 01 15:40:35.960924 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:35.960895    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" path="/var/lib/kubelet/pods/bbb5f2ae-d498-4cde-95be-e7dd1339c347/volumes"
Nov 01 15:40:43.581154 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:43.581118    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:40:44.904576 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:44.904533    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main" containerID="cri-o://70f45ac23fc38002775042400ec16e6fc2b8c4943cec90791b71d61ee23a2e82" gracePeriod=45
Nov 01 15:40:45.143273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.143142    5423 generic.go:356] "Generic (PLEG): container finished" podID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerID="70f45ac23fc38002775042400ec16e6fc2b8c4943cec90791b71d61ee23a2e82" exitCode=0
Nov 01 15:40:45.143273 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.143214    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" event={"ID":"29709b9a-d3f9-4795-a58a-9b2ba35950fb","Type":"ContainerDied","Data":"70f45ac23fc38002775042400ec16e6fc2b8c4943cec90791b71d61ee23a2e82"}
Nov 01 15:40:45.207113 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.207089    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 15:40:45.262661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.262623    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tls-certs\") pod \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") "
Nov 01 15:40:45.262832 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.262667    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29709b9a-d3f9-4795-a58a-9b2ba35950fb-kube-api-access-nxsn5\" (UniqueName: \"kubernetes.io/projected/29709b9a-d3f9-4795-a58a-9b2ba35950fb-kube-api-access-nxsn5\") pod \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") "
Nov 01 15:40:45.262832 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.262734    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-model-cache\") pod \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") "
Nov 01 15:40:45.262960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.262842    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-home\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-home\") pod \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") "
Nov 01 15:40:45.263028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.262969    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tmp-dir\") pod \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") "
Nov 01 15:40:45.263028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.263016    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-dshm\") pod \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\" (UID: \"29709b9a-d3f9-4795-a58a-9b2ba35950fb\") "
Nov 01 15:40:45.263132 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.263043    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-model-cache" pod "29709b9a-d3f9-4795-a58a-9b2ba35950fb" (UID: "29709b9a-d3f9-4795-a58a-9b2ba35950fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.263132 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.263079    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-home" pod "29709b9a-d3f9-4795-a58a-9b2ba35950fb" (UID: "29709b9a-d3f9-4795-a58a-9b2ba35950fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.263226 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.263173    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tmp-dir" pod "29709b9a-d3f9-4795-a58a-9b2ba35950fb" (UID: "29709b9a-d3f9-4795-a58a-9b2ba35950fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.263589 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.263566    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.263589 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.263588    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.263795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.263599    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.264978 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.264953    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tls-certs" pod "29709b9a-d3f9-4795-a58a-9b2ba35950fb" (UID: "29709b9a-d3f9-4795-a58a-9b2ba35950fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:45.265090 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.264988    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29709b9a-d3f9-4795-a58a-9b2ba35950fb-kube-api-access-nxsn5" pod "29709b9a-d3f9-4795-a58a-9b2ba35950fb" (UID: "29709b9a-d3f9-4795-a58a-9b2ba35950fb"). InnerVolumeSpecName "kube-api-access-nxsn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:45.265090 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.265007    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-dshm" pod "29709b9a-d3f9-4795-a58a-9b2ba35950fb" (UID: "29709b9a-d3f9-4795-a58a-9b2ba35950fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.364776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.364744    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29709b9a-d3f9-4795-a58a-9b2ba35950fb-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.364776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.364771    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29709b9a-d3f9-4795-a58a-9b2ba35950fb-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.364776 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:45.364782    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxsn5\" (UniqueName: \"kubernetes.io/projected/29709b9a-d3f9-4795-a58a-9b2ba35950fb-kube-api-access-nxsn5\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:46.150073 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:46.150040    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"
Nov 01 15:40:46.150575 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:46.150071    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2" event={"ID":"29709b9a-d3f9-4795-a58a-9b2ba35950fb","Type":"ContainerDied","Data":"5c8a3714e88d5d7888b4e4c413e5fc08b87283e9449dda6d8135a4b6af240632"}
Nov 01 15:40:46.150575 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:46.150123    5423 scope.go:122] "RemoveContainer" containerID="70f45ac23fc38002775042400ec16e6fc2b8c4943cec90791b71d61ee23a2e82"
Nov 01 15:40:46.316978 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:46.316943    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"]
Nov 01 15:40:46.319988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:46.319969    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/simulator-w4-536a87-kserve-7ff878867-k58k2"]
Nov 01 15:40:47.960827 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:47.960797    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" path="/var/lib/kubelet/pods/29709b9a-d3f9-4795-a58a-9b2ba35950fb/volumes"
Nov 01 15:40:58.778795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:58.778755    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"]
Nov 01 15:40:58.779494 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:58.778956    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk" podUID="fe198337-d3c3-4aed-a9d3-8457fcfed95c" containerName="maas-api" containerID="cri-o://2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b" gracePeriod=30
Nov 01 15:40:59.037734 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.037676    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 15:40:59.112491 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.112459    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls\") pod \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") "
Nov 01 15:40:59.112656 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.112533    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe198337-d3c3-4aed-a9d3-8457fcfed95c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe198337-d3c3-4aed-a9d3-8457fcfed95c-openshift-service-ca\") pod \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") "
Nov 01 15:40:59.112656 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.112628    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-metrics-certs\") pod \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") "
Nov 01 15:40:59.112778 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.112766    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe198337-d3c3-4aed-a9d3-8457fcfed95c-kube-api-access-8st4b\" (UniqueName: \"kubernetes.io/projected/fe198337-d3c3-4aed-a9d3-8457fcfed95c-kube-api-access-8st4b\") pod \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\" (UID: \"fe198337-d3c3-4aed-a9d3-8457fcfed95c\") "
Nov 01 15:40:59.112902 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.112866    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe198337-d3c3-4aed-a9d3-8457fcfed95c-openshift-service-ca" pod "fe198337-d3c3-4aed-a9d3-8457fcfed95c" (UID: "fe198337-d3c3-4aed-a9d3-8457fcfed95c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:40:59.113275 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.113231    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe198337-d3c3-4aed-a9d3-8457fcfed95c-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:59.114800 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.114773    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls" pod "fe198337-d3c3-4aed-a9d3-8457fcfed95c" (UID: "fe198337-d3c3-4aed-a9d3-8457fcfed95c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:59.114901 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.114800    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe198337-d3c3-4aed-a9d3-8457fcfed95c-kube-api-access-8st4b" pod "fe198337-d3c3-4aed-a9d3-8457fcfed95c" (UID: "fe198337-d3c3-4aed-a9d3-8457fcfed95c"). InnerVolumeSpecName "kube-api-access-8st4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:59.114901 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.114820    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-metrics-certs" pod "fe198337-d3c3-4aed-a9d3-8457fcfed95c" (UID: "fe198337-d3c3-4aed-a9d3-8457fcfed95c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:59.214379 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.214348    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:59.214379 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.214376    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe198337-d3c3-4aed-a9d3-8457fcfed95c-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:59.214608 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.214407    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8st4b\" (UniqueName: \"kubernetes.io/projected/fe198337-d3c3-4aed-a9d3-8457fcfed95c-kube-api-access-8st4b\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:59.225964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.225933    5423 generic.go:356] "Generic (PLEG): container finished" podID="fe198337-d3c3-4aed-a9d3-8457fcfed95c" containerID="2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b" exitCode=0
Nov 01 15:40:59.226115 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.225984    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"
Nov 01 15:40:59.226115 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.226027    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk" event={"ID":"fe198337-d3c3-4aed-a9d3-8457fcfed95c","Type":"ContainerDied","Data":"2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b"}
Nov 01 15:40:59.226115 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.226086    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk" event={"ID":"fe198337-d3c3-4aed-a9d3-8457fcfed95c","Type":"ContainerDied","Data":"cdee74c43963e461eb46d714df301a83978bf5b22dae4bb99e79aaa66fdd5a03"}
Nov 01 15:40:59.226115 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.226114    5423 scope.go:122] "RemoveContainer" containerID="2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b"
Nov 01 15:40:59.237031 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.237006    5423 scope.go:122] "RemoveContainer" containerID="2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b"
Nov 01 15:40:59.237311 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:40:59.237288    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b\": container with ID starting with 2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b not found: ID does not exist" containerID="2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b"
Nov 01 15:40:59.237453 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.237317    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b"} err="failed to get container status \"2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b\": rpc error: code = NotFound desc = could not find container \"2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b\": container with ID starting with 2ebc8b33aeabe5e544dbdc2aa60706ed8bf0199c2ad8cb1c0e89d9279d844d2b not found: ID does not exist"
Nov 01 15:40:59.397996 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.397960    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"]
Nov 01 15:40:59.403681 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.403658    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-536a87-69785c9797-gskvk"]
Nov 01 15:40:59.961713 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:40:59.961684    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe198337-d3c3-4aed-a9d3-8457fcfed95c" path="/var/lib/kubelet/pods/fe198337-d3c3-4aed-a9d3-8457fcfed95c/volumes"
Nov 01 15:41:05.149872 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:41:05.149831    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:41:13.144960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:13.144919    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"]
Nov 01 15:41:13.195769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:13.195740    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"]
Nov 01 15:41:14.907153 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:14.907114    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"]
Nov 01 15:41:14.907586 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:14.907364    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc" podUID="081a7f84-b0fc-46da-afbd-9b58fe9c6322" containerName="maas-api" containerID="cri-o://fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6" gracePeriod=30
Nov 01 15:41:15.220699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.220673    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:41:15.324215 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.324183    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081a7f84-b0fc-46da-afbd-9b58fe9c6322-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081a7f84-b0fc-46da-afbd-9b58fe9c6322-openshift-service-ca\") pod \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") "
Nov 01 15:41:15.324433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.324247    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-metrics-certs\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-metrics-certs\") pod \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") "
Nov 01 15:41:15.324433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.324345    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081a7f84-b0fc-46da-afbd-9b58fe9c6322-kube-api-access-5csfr\" (UniqueName: \"kubernetes.io/projected/081a7f84-b0fc-46da-afbd-9b58fe9c6322-kube-api-access-5csfr\") pod \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") "
Nov 01 15:41:15.324433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.324422    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls\") pod \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\" (UID: \"081a7f84-b0fc-46da-afbd-9b58fe9c6322\") "
Nov 01 15:41:15.324633 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.324578    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081a7f84-b0fc-46da-afbd-9b58fe9c6322-openshift-service-ca" pod "081a7f84-b0fc-46da-afbd-9b58fe9c6322" (UID: "081a7f84-b0fc-46da-afbd-9b58fe9c6322"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:41:15.324906 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.324885    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081a7f84-b0fc-46da-afbd-9b58fe9c6322-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.326915 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.326837    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081a7f84-b0fc-46da-afbd-9b58fe9c6322-kube-api-access-5csfr" pod "081a7f84-b0fc-46da-afbd-9b58fe9c6322" (UID: "081a7f84-b0fc-46da-afbd-9b58fe9c6322"). InnerVolumeSpecName "kube-api-access-5csfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:15.327040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.326987    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-metrics-certs" pod "081a7f84-b0fc-46da-afbd-9b58fe9c6322" (UID: "081a7f84-b0fc-46da-afbd-9b58fe9c6322"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:15.327107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.327041    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls" pod "081a7f84-b0fc-46da-afbd-9b58fe9c6322" (UID: "081a7f84-b0fc-46da-afbd-9b58fe9c6322"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:15.332084 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.332056    5423 generic.go:356] "Generic (PLEG): container finished" podID="081a7f84-b0fc-46da-afbd-9b58fe9c6322" containerID="fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6" exitCode=0
Nov 01 15:41:15.332212 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.332098    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"
Nov 01 15:41:15.332212 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.332138    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc" event={"ID":"081a7f84-b0fc-46da-afbd-9b58fe9c6322","Type":"ContainerDied","Data":"fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6"}
Nov 01 15:41:15.332212 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.332196    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc" event={"ID":"081a7f84-b0fc-46da-afbd-9b58fe9c6322","Type":"ContainerDied","Data":"ff66a13e3749ac5bbf9ad9f959dba9afa44f26b7d04ff5da0f1457707c7db6c0"}
Nov 01 15:41:15.332451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.332227    5423 scope.go:122] "RemoveContainer" containerID="fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6"
Nov 01 15:41:15.360014 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.359987    5423 scope.go:122] "RemoveContainer" containerID="fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6"
Nov 01 15:41:15.360398 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:41:15.360351    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6\": container with ID starting with fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6 not found: ID does not exist" containerID="fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6"
Nov 01 15:41:15.360532 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.360416    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6"} err="failed to get container status \"fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6\": rpc error: code = NotFound desc = could not find container \"fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6\": container with ID starting with fb71fad6d1e27dd258176f35f1488764c51b3ed75e56f2f830c1b8a5fe69bba6 not found: ID does not exist"
Nov 01 15:41:15.425607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.425569    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.425607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.425599    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5csfr\" (UniqueName: \"kubernetes.io/projected/081a7f84-b0fc-46da-afbd-9b58fe9c6322-kube-api-access-5csfr\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.425607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.425610    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081a7f84-b0fc-46da-afbd-9b58fe9c6322-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.522578 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.522492    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"]
Nov 01 15:41:15.529446 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.529408    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55943a-85d8b67d59-p4lxc"]
Nov 01 15:41:15.961116 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:15.961079    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081a7f84-b0fc-46da-afbd-9b58fe9c6322" path="/var/lib/kubelet/pods/081a7f84-b0fc-46da-afbd-9b58fe9c6322/volumes"
Nov 01 15:41:28.165617 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.165578    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main" containerID="cri-o://37ea0e83332f984730073fe1bf99fd224d6a3125d0635c44581683bfbf2d852d" gracePeriod=45
Nov 01 15:41:28.215588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.215548    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main" containerID="cri-o://2ed59a259d2cf240cd512140e3ad0090027d4b3a3be230d321aa24358680bc66" gracePeriod=45
Nov 01 15:41:28.428024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.427944    5423 generic.go:356] "Generic (PLEG): container finished" podID="85e30a1b-095a-450b-98af-ecae621e1047" containerID="2ed59a259d2cf240cd512140e3ad0090027d4b3a3be230d321aa24358680bc66" exitCode=0
Nov 01 15:41:28.428168 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.428051    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" event={"ID":"85e30a1b-095a-450b-98af-ecae621e1047","Type":"ContainerDied","Data":"2ed59a259d2cf240cd512140e3ad0090027d4b3a3be230d321aa24358680bc66"}
Nov 01 15:41:28.430767 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.430743    5423 generic.go:356] "Generic (PLEG): container finished" podID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerID="37ea0e83332f984730073fe1bf99fd224d6a3125d0635c44581683bfbf2d852d" exitCode=0
Nov 01 15:41:28.430913 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.430823    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" event={"ID":"6a367b54-8394-4670-9a95-0ad8e9e08c0c","Type":"ContainerDied","Data":"37ea0e83332f984730073fe1bf99fd224d6a3125d0635c44581683bfbf2d852d"}
Nov 01 15:41:28.489341 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.489312    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 15:41:28.497562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.497543    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 15:41:28.600929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.600891    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tmp-dir\") pod \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") "
Nov 01 15:41:28.601127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.600948    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-home\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-home\") pod \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") "
Nov 01 15:41:28.601127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.600987    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tls-certs\") pod \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") "
Nov 01 15:41:28.601127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601108    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-dshm\") pod \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") "
Nov 01 15:41:28.601317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601138    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a367b54-8394-4670-9a95-0ad8e9e08c0c-kube-api-access-86784\" (UniqueName: \"kubernetes.io/projected/6a367b54-8394-4670-9a95-0ad8e9e08c0c-kube-api-access-86784\") pod \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") "
Nov 01 15:41:28.601317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601176    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-tmp-dir\") pod \"85e30a1b-095a-450b-98af-ecae621e1047\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") "
Nov 01 15:41:28.601317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601211    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e30a1b-095a-450b-98af-ecae621e1047-tls-certs\" (UniqueName: \"kubernetes.io/secret/85e30a1b-095a-450b-98af-ecae621e1047-tls-certs\") pod \"85e30a1b-095a-450b-98af-ecae621e1047\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") "
Nov 01 15:41:28.601317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601210    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-home" pod "6a367b54-8394-4670-9a95-0ad8e9e08c0c" (UID: "6a367b54-8394-4670-9a95-0ad8e9e08c0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.601317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601302    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e30a1b-095a-450b-98af-ecae621e1047-kube-api-access-n4drw\" (UniqueName: \"kubernetes.io/projected/85e30a1b-095a-450b-98af-ecae621e1047-kube-api-access-n4drw\") pod \"85e30a1b-095a-450b-98af-ecae621e1047\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") "
Nov 01 15:41:28.601626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601344    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-model-cache\") pod \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\" (UID: \"6a367b54-8394-4670-9a95-0ad8e9e08c0c\") "
Nov 01 15:41:28.601626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601551    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-home\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-home\") pod \"85e30a1b-095a-450b-98af-ecae621e1047\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") "
Nov 01 15:41:28.601736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601703    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-model-cache\") pod \"85e30a1b-095a-450b-98af-ecae621e1047\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") "
Nov 01 15:41:28.601792 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601746    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-dshm\") pod \"85e30a1b-095a-450b-98af-ecae621e1047\" (UID: \"85e30a1b-095a-450b-98af-ecae621e1047\") "
Nov 01 15:41:28.601792 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.601744    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tmp-dir" pod "6a367b54-8394-4670-9a95-0ad8e9e08c0c" (UID: "6a367b54-8394-4670-9a95-0ad8e9e08c0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.602039 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.602016    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-home" pod "85e30a1b-095a-450b-98af-ecae621e1047" (UID: "85e30a1b-095a-450b-98af-ecae621e1047"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.602851 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.602337    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.602851 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.602364    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.602851 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.602378    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.602851 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.602500    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-tmp-dir" pod "85e30a1b-095a-450b-98af-ecae621e1047" (UID: "85e30a1b-095a-450b-98af-ecae621e1047"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.602851 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.602648    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-model-cache" pod "85e30a1b-095a-450b-98af-ecae621e1047" (UID: "85e30a1b-095a-450b-98af-ecae621e1047"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.602851 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.602824    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-model-cache" pod "6a367b54-8394-4670-9a95-0ad8e9e08c0c" (UID: "6a367b54-8394-4670-9a95-0ad8e9e08c0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.604172 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.604151    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tls-certs" pod "6a367b54-8394-4670-9a95-0ad8e9e08c0c" (UID: "6a367b54-8394-4670-9a95-0ad8e9e08c0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:28.604290 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.604175    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a367b54-8394-4670-9a95-0ad8e9e08c0c-kube-api-access-86784" pod "6a367b54-8394-4670-9a95-0ad8e9e08c0c" (UID: "6a367b54-8394-4670-9a95-0ad8e9e08c0c"). InnerVolumeSpecName "kube-api-access-86784". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:28.604615 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.604590    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-dshm" pod "85e30a1b-095a-450b-98af-ecae621e1047" (UID: "85e30a1b-095a-450b-98af-ecae621e1047"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.606451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.606417    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e30a1b-095a-450b-98af-ecae621e1047-tls-certs" pod "85e30a1b-095a-450b-98af-ecae621e1047" (UID: "85e30a1b-095a-450b-98af-ecae621e1047"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:28.606692 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.606645    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e30a1b-095a-450b-98af-ecae621e1047-kube-api-access-n4drw" pod "85e30a1b-095a-450b-98af-ecae621e1047" (UID: "85e30a1b-095a-450b-98af-ecae621e1047"). InnerVolumeSpecName "kube-api-access-n4drw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:28.606810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.606708    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-dshm" pod "6a367b54-8394-4670-9a95-0ad8e9e08c0c" (UID: "6a367b54-8394-4670-9a95-0ad8e9e08c0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.703373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703284    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.703373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703323    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a367b54-8394-4670-9a95-0ad8e9e08c0c-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.703373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703338    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.703373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703350    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86784\" (UniqueName: \"kubernetes.io/projected/6a367b54-8394-4670-9a95-0ad8e9e08c0c-kube-api-access-86784\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.703373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703361    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.703373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703370    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e30a1b-095a-450b-98af-ecae621e1047-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.703373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703380    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4drw\" (UniqueName: \"kubernetes.io/projected/85e30a1b-095a-450b-98af-ecae621e1047-kube-api-access-n4drw\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.703964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703416    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a367b54-8394-4670-9a95-0ad8e9e08c0c-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.703964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:28.703427    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e30a1b-095a-450b-98af-ecae621e1047-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:29.438328 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.438251    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"
Nov 01 15:41:29.438811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.438246    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9" event={"ID":"85e30a1b-095a-450b-98af-ecae621e1047","Type":"ContainerDied","Data":"5ec13050b0e208467c0b3b103ae94991a3ef93f95134ac64982b233971f59e65"}
Nov 01 15:41:29.438811 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.438402    5423 scope.go:122] "RemoveContainer" containerID="2ed59a259d2cf240cd512140e3ad0090027d4b3a3be230d321aa24358680bc66"
Nov 01 15:41:29.439863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.439711    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8" event={"ID":"6a367b54-8394-4670-9a95-0ad8e9e08c0c","Type":"ContainerDied","Data":"ef433579c1e2162e9e69d05fdd8c1d1c5c79c538f2aff74f2ad09412fee6bfbe"}
Nov 01 15:41:29.439863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.439734    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"
Nov 01 15:41:29.447907 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.447887    5423 scope.go:122] "RemoveContainer" containerID="37ea0e83332f984730073fe1bf99fd224d6a3125d0635c44581683bfbf2d852d"
Nov 01 15:41:29.603472 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.603439    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"]
Nov 01 15:41:29.606950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.606925    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-2-w2-007270-kserve-747bdf4bcd-924h8"]
Nov 01 15:41:29.717272 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.717209    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"]
Nov 01 15:41:29.723183 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.723158    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-007270/distinct-w2-007270-kserve-84bfb5d9c-4l4c9"]
Nov 01 15:41:29.964549 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.964518    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" path="/var/lib/kubelet/pods/6a367b54-8394-4670-9a95-0ad8e9e08c0c/volumes"
Nov 01 15:41:29.965165 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:29.965142    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e30a1b-095a-450b-98af-ecae621e1047" path="/var/lib/kubelet/pods/85e30a1b-095a-450b-98af-ecae621e1047/volumes"
Nov 01 15:41:30.777491 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:30.777449    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"]
Nov 01 15:41:30.875951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:30.875915    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"]
Nov 01 15:41:35.150275 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:41:35.150234    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:41:35.778016 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:35.777974    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv" podUID="727d2dcc-9e55-4a28-8f6b-85510fb014fd" containerName="payload-processing" containerID="cri-o://2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28" gracePeriod=25
Nov 01 15:41:35.876842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:35.876797    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b" podUID="e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" containerName="payload-pre-processing" containerID="cri-o://beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c" gracePeriod=25
Nov 01 15:41:36.038144 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.038079    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:41:36.186751 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.186715    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727d2dcc-9e55-4a28-8f6b-85510fb014fd-kube-api-access-6bzlc\" (UniqueName: \"kubernetes.io/projected/727d2dcc-9e55-4a28-8f6b-85510fb014fd-kube-api-access-6bzlc\") pod \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\" (UID: \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\") "
Nov 01 15:41:36.187123 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.186965    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727d2dcc-9e55-4a28-8f6b-85510fb014fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727d2dcc-9e55-4a28-8f6b-85510fb014fd-plugins-config-volume\") pod \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\" (UID: \"727d2dcc-9e55-4a28-8f6b-85510fb014fd\") "
Nov 01 15:41:36.187453 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.187425    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727d2dcc-9e55-4a28-8f6b-85510fb014fd-plugins-config-volume" pod "727d2dcc-9e55-4a28-8f6b-85510fb014fd" (UID: "727d2dcc-9e55-4a28-8f6b-85510fb014fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:41:36.189025 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.188998    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727d2dcc-9e55-4a28-8f6b-85510fb014fd-kube-api-access-6bzlc" pod "727d2dcc-9e55-4a28-8f6b-85510fb014fd" (UID: "727d2dcc-9e55-4a28-8f6b-85510fb014fd"). InnerVolumeSpecName "kube-api-access-6bzlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:36.287980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.287954    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bzlc\" (UniqueName: \"kubernetes.io/projected/727d2dcc-9e55-4a28-8f6b-85510fb014fd-kube-api-access-6bzlc\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:36.287980 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.287978    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727d2dcc-9e55-4a28-8f6b-85510fb014fd-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:36.487367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.487329    5423 generic.go:356] "Generic (PLEG): container finished" podID="727d2dcc-9e55-4a28-8f6b-85510fb014fd" containerID="2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28" exitCode=0
Nov 01 15:41:36.487731 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.487510    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"
Nov 01 15:41:36.487933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.487533    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv" event={"ID":"727d2dcc-9e55-4a28-8f6b-85510fb014fd","Type":"ContainerDied","Data":"2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28"}
Nov 01 15:41:36.488129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.488108    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv" event={"ID":"727d2dcc-9e55-4a28-8f6b-85510fb014fd","Type":"ContainerDied","Data":"e62cde35760ba02ed560aa4af996019585d7de355762635b58145589ef370c73"}
Nov 01 15:41:36.488526 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.488237    5423 scope.go:122] "RemoveContainer" containerID="2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28"
Nov 01 15:41:36.516266 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.516241    5423 scope.go:122] "RemoveContainer" containerID="2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28"
Nov 01 15:41:36.516674 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:41:36.516648    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28\": container with ID starting with 2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28 not found: ID does not exist" containerID="2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28"
Nov 01 15:41:36.516753 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.516685    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28"} err="failed to get container status \"2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28\": rpc error: code = NotFound desc = could not find container \"2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28\": container with ID starting with 2f7769ffc66cbe71c5f591e8b5ec3b92a64f96383f20e277ad97bf7d66853b28 not found: ID does not exist"
Nov 01 15:41:36.674677 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.674639    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"]
Nov 01 15:41:36.674885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:36.674703    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3f45c3-6cc477d795-4b4kv"]
Nov 01 15:41:37.960260 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:37.960229    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727d2dcc-9e55-4a28-8f6b-85510fb014fd" path="/var/lib/kubelet/pods/727d2dcc-9e55-4a28-8f6b-85510fb014fd/volumes"
Nov 01 15:41:38.751139 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:38.751094    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b" podUID="e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:41:46.290219 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:46.290183    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:41:48.751262 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:48.751222    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b" podUID="e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:41:56.121751 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.121725    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:41:56.211663 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.211582    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-kube-api-access-fct82\" (UniqueName: \"kubernetes.io/projected/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-kube-api-access-fct82\") pod \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\" (UID: \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\") "
Nov 01 15:41:56.211827 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.211754    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-plugins-config-volume\") pod \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\" (UID: \"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143\") "
Nov 01 15:41:56.212197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.212169    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-plugins-config-volume" pod "e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" (UID: "e7a0e0fa-f4e9-49d3-abcf-67c6cd089143"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:41:56.213785 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.213753    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-kube-api-access-fct82" pod "e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" (UID: "e7a0e0fa-f4e9-49d3-abcf-67c6cd089143"). InnerVolumeSpecName "kube-api-access-fct82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:56.312965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.312926    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:56.312965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.312958    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fct82\" (UniqueName: \"kubernetes.io/projected/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143-kube-api-access-fct82\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:56.602593 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.602562    5423 generic.go:356] "Generic (PLEG): container finished" podID="e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" containerID="beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c" exitCode=0
Nov 01 15:41:56.602787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.602608    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"
Nov 01 15:41:56.602787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.602646    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b" event={"ID":"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143","Type":"ContainerDied","Data":"beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c"}
Nov 01 15:41:56.602787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.602702    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b" event={"ID":"e7a0e0fa-f4e9-49d3-abcf-67c6cd089143","Type":"ContainerDied","Data":"6634769ae14263bf0850372194d7a47a5fea1f56e4d1ed720ac66e3fa62d4a9c"}
Nov 01 15:41:56.602787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.602728    5423 scope.go:122] "RemoveContainer" containerID="beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c"
Nov 01 15:41:56.614470 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.614450    5423 scope.go:122] "RemoveContainer" containerID="beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c"
Nov 01 15:41:56.614901 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:41:56.614873    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c\": container with ID starting with beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c not found: ID does not exist" containerID="beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c"
Nov 01 15:41:56.615000 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.614908    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c"} err="failed to get container status \"beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c\": rpc error: code = NotFound desc = could not find container \"beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c\": container with ID starting with beb74d152570698fac4d798f7f8e92dccffc419eb402c2c5482956741e83d59c not found: ID does not exist"
Nov 01 15:41:56.769310 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.769275    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"]
Nov 01 15:41:56.819875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:56.819839    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3f45c3-659c7778ff-x7s6b"]
Nov 01 15:41:57.960009 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:41:57.959979    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" path="/var/lib/kubelet/pods/e7a0e0fa-f4e9-49d3-abcf-67c6cd089143/volumes"
Nov 01 15:42:05.148135 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:05.148099    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:42:10.410320 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:10.410291    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:42:13.134497 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:13.134466    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"]
Nov 01 15:42:13.134897 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:13.134683    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy" containerID="cri-o://2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca" gracePeriod=30
Nov 01 15:42:14.410583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:14.410551    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"]
Nov 01 15:42:14.410949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:14.410775    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy" containerID="cri-o://8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694" gracePeriod=30
Nov 01 15:42:18.392367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.392339    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:42:18.554178 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554111    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-credential-socket\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.554178 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554166    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-envoy\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.554434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554191    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-token\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.554434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554234    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-socket\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.554434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554281    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-data\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.554620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554436    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-kube-api-access-2db62\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-kube-api-access-2db62\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.554620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554498    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-certs\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.554620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554436    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-credential-socket" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:18.554620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554558    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-data" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:18.554620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554567    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-podinfo\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.554620 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554577    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-socket" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:18.555044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554653    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74a9078-63f9-4173-bbe5-e893254c0f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74a9078-63f9-4173-bbe5-e893254c0f8d-istiod-ca-cert\") pod \"e74a9078-63f9-4173-bbe5-e893254c0f8d\" (UID: \"e74a9078-63f9-4173-bbe5-e893254c0f8d\") "
Nov 01 15:42:18.555044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554773    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-certs" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:18.555044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.554969    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74a9078-63f9-4173-bbe5-e893254c0f8d-istiod-ca-cert" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:18.555218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.555165    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74a9078-63f9-4173-bbe5-e893254c0f8d-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.555218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.555187    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.555218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.555203    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.555418 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.555217    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.555418 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.555232    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.556524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.556494    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-envoy" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:18.556643 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.556561    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-token" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:18.556643 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.556592    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-kube-api-access-2db62" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "kube-api-access-2db62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:18.556731 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.556716    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-podinfo" pod "e74a9078-63f9-4173-bbe5-e893254c0f8d" (UID: "e74a9078-63f9-4173-bbe5-e893254c0f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:42:18.656764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.656733    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.656764 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.656761    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.656923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.656775    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.656923 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.656788    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2db62\" (UniqueName: \"kubernetes.io/projected/e74a9078-63f9-4173-bbe5-e893254c0f8d-kube-api-access-2db62\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:18.718431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.718381    5423 generic.go:356] "Generic (PLEG): container finished" podID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerID="2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca" exitCode=0
Nov 01 15:42:18.718558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.718462    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"
Nov 01 15:42:18.718558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.718484    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" event={"ID":"e74a9078-63f9-4173-bbe5-e893254c0f8d","Type":"ContainerDied","Data":"2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca"}
Nov 01 15:42:18.718558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.718539    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2" event={"ID":"e74a9078-63f9-4173-bbe5-e893254c0f8d","Type":"ContainerDied","Data":"87d2ebe5703c3b82f8c36303ecbab23d7862815a728a5d732b2ef0d31039a1fa"}
Nov 01 15:42:18.718729 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.718561    5423 scope.go:122] "RemoveContainer" containerID="2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca"
Nov 01 15:42:18.728237 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.728221    5423 scope.go:122] "RemoveContainer" containerID="fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36"
Nov 01 15:42:18.736232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.736217    5423 scope.go:122] "RemoveContainer" containerID="2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca"
Nov 01 15:42:18.736498 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:18.736481    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca\": container with ID starting with 2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca not found: ID does not exist" containerID="2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca"
Nov 01 15:42:18.736589 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.736510    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca"} err="failed to get container status \"2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca\": rpc error: code = NotFound desc = could not find container \"2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca\": container with ID starting with 2c69a4db98a8ae590afc10aeb1b7d4c9a1b1959024b28d278a960c821d9b84ca not found: ID does not exist"
Nov 01 15:42:18.736589 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.736533    5423 scope.go:122] "RemoveContainer" containerID="fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36"
Nov 01 15:42:18.736811 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:18.736791    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36\": container with ID starting with fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36 not found: ID does not exist" containerID="fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36"
Nov 01 15:42:18.736890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.736819    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36"} err="failed to get container status \"fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36\": rpc error: code = NotFound desc = could not find container \"fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36\": container with ID starting with fafae44f7d1bbced4d2a366bf3a8af26d5d51f566c3f82e8f56d4279b5f42e36 not found: ID does not exist"
Nov 01 15:42:18.886329 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.886291    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"]
Nov 01 15:42:18.890178 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:18.890152    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3f45c3-openshift-default-7c7cb989c-fnfk2"]
Nov 01 15:42:19.670875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.670852    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:42:19.726816 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.726740    5423 generic.go:356] "Generic (PLEG): container finished" podID="520751d7-df0b-4875-8e79-9d9703298a93" containerID="8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694" exitCode=0
Nov 01 15:42:19.726816 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.726782    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"
Nov 01 15:42:19.727017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.726820    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" event={"ID":"520751d7-df0b-4875-8e79-9d9703298a93","Type":"ContainerDied","Data":"8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694"}
Nov 01 15:42:19.727017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.726904    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg" event={"ID":"520751d7-df0b-4875-8e79-9d9703298a93","Type":"ContainerDied","Data":"79d21d91b991d14df30719b961d78790360023ace6045b761cdc6ce91fc7bd21"}
Nov 01 15:42:19.727017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.726932    5423 scope.go:122] "RemoveContainer" containerID="8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694"
Nov 01 15:42:19.736019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.736002    5423 scope.go:122] "RemoveContainer" containerID="7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29"
Nov 01 15:42:19.744417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.744374    5423 scope.go:122] "RemoveContainer" containerID="8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694"
Nov 01 15:42:19.744690 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:19.744671    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694\": container with ID starting with 8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694 not found: ID does not exist" containerID="8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694"
Nov 01 15:42:19.744736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.744698    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694"} err="failed to get container status \"8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694\": rpc error: code = NotFound desc = could not find container \"8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694\": container with ID starting with 8231b86c9b5c5ec362b339e71edad26b7b5f44237ec1466015ad45ee3549e694 not found: ID does not exist"
Nov 01 15:42:19.744736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.744715    5423 scope.go:122] "RemoveContainer" containerID="7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29"
Nov 01 15:42:19.744966 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:19.744948    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29\": container with ID starting with 7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29 not found: ID does not exist" containerID="7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29"
Nov 01 15:42:19.745069 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.744972    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29"} err="failed to get container status \"7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29\": rpc error: code = NotFound desc = could not find container \"7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29\": container with ID starting with 7d0de4a0d5857bb3d812c063db2e3931c6f7c97c68750b0b9bb95ec7a1da5a29 not found: ID does not exist"
Nov 01 15:42:19.766314 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766293    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/520751d7-df0b-4875-8e79-9d9703298a93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520751d7-df0b-4875-8e79-9d9703298a93-istio-podinfo\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766332    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-socket\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766363    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-envoy\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766402    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520751d7-df0b-4875-8e79-9d9703298a93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520751d7-df0b-4875-8e79-9d9703298a93-istiod-ca-cert\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766617 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766541    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-credential-socket\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766617 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766607    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-certs\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766730 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766655    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-kube-api-access-8qtd2\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-kube-api-access-8qtd2\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766730 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766666    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-socket" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:19.766730 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766689    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-data\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766757    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-istio-token\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-istio-token\") pod \"520751d7-df0b-4875-8e79-9d9703298a93\" (UID: \"520751d7-df0b-4875-8e79-9d9703298a93\") "
Nov 01 15:42:19.766890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766826    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-certs" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:19.766890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766841    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-credential-socket" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:19.766890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.766858    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520751d7-df0b-4875-8e79-9d9703298a93-istiod-ca-cert" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:19.767217 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.767194    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.767335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.767232    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520751d7-df0b-4875-8e79-9d9703298a93-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.767335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.767252    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.767335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.767265    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.767335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.767194    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-data" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:19.769080 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.769042    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/520751d7-df0b-4875-8e79-9d9703298a93-istio-podinfo" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:42:19.769315 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.769293    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-envoy" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:19.769480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.769462    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-kube-api-access-8qtd2" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "kube-api-access-8qtd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:19.769553 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.769489    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-istio-token" pod "520751d7-df0b-4875-8e79-9d9703298a93" (UID: "520751d7-df0b-4875-8e79-9d9703298a93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:19.868009 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.867975    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520751d7-df0b-4875-8e79-9d9703298a93-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.868009 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.868007    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.868261 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.868021    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qtd2\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-kube-api-access-8qtd2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.868261 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.868036    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520751d7-df0b-4875-8e79-9d9703298a93-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.868261 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.868049    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520751d7-df0b-4875-8e79-9d9703298a93-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:19.962336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:19.962300    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" path="/var/lib/kubelet/pods/e74a9078-63f9-4173-bbe5-e893254c0f8d/volumes"
Nov 01 15:42:20.143182 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:20.143151    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"]
Nov 01 15:42:20.155757 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:20.155715    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-af2dd2-openshift-default-fcbfc9c5c-fnmkg"]
Nov 01 15:42:21.317045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:21.317009    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"]
Nov 01 15:42:21.317467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:21.317211    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy" containerID="cri-o://3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f" gracePeriod=30
Nov 01 15:42:21.659299 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:21.659208    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"]
Nov 01 15:42:21.962678 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:21.962609    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520751d7-df0b-4875-8e79-9d9703298a93" path="/var/lib/kubelet/pods/520751d7-df0b-4875-8e79-9d9703298a93/volumes"
Nov 01 15:42:23.891287 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:23.891253    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"]
Nov 01 15:42:23.997339 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:23.997308    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"]
Nov 01 15:42:26.562010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.561984    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 15:42:26.626588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626506    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-certs\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.626588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626567    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-socket\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.626821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626603    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-podinfo\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.626821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626650    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istiod-ca-cert\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.626821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626686    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-envoy\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.626821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626727    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-kube-api-access-8vnt7\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-kube-api-access-8vnt7\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.626821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626748    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-certs" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:26.627103 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626821    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-token\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-token\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.627103 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626832    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-socket" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:26.627103 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626872    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-credential-socket\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.627103 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.626910    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-data\") pod \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\" (UID: \"61f5dd4e-c04a-47ec-8291-5a6565e93bde\") "
Nov 01 15:42:26.627103 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.627088    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istiod-ca-cert" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:26.627687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.627357    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-credential-socket" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:26.627687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.627596    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.627687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.627631    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.627687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.627648    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.627687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.627663    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.627687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.627660    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-data" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:26.629091 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.629060    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-podinfo" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:42:26.629179 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.629098    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-envoy" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:42:26.629370 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.629352    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-token" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:26.629370 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.629360    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-kube-api-access-8vnt7" pod "61f5dd4e-c04a-47ec-8291-5a6565e93bde" (UID: "61f5dd4e-c04a-47ec-8291-5a6565e93bde"). InnerVolumeSpecName "kube-api-access-8vnt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:26.660235 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.660202    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv" podUID="4c209a58-3ede-47f3-81dd-813a0ba66507" containerName="payload-processing" containerID="cri-o://612a176f3c39252dd6135fb0641512a4105dab27d31624ce5bde6c187d0f31ef" gracePeriod=25
Nov 01 15:42:26.728893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.728861    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.728893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.728887    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.728893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.728897    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.729109 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.728905    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vnt7\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-kube-api-access-8vnt7\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.729109 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.728916    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f5dd4e-c04a-47ec-8291-5a6565e93bde-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:26.771348 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.771311    5423 generic.go:356] "Generic (PLEG): container finished" podID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerID="3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f" exitCode=0
Nov 01 15:42:26.771542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.771382    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"
Nov 01 15:42:26.771542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.771419    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" event={"ID":"61f5dd4e-c04a-47ec-8291-5a6565e93bde","Type":"ContainerDied","Data":"3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f"}
Nov 01 15:42:26.771542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.771473    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj" event={"ID":"61f5dd4e-c04a-47ec-8291-5a6565e93bde","Type":"ContainerDied","Data":"37528bcbdd3456a7c62874c2d4930f261c746ff278b11065c11a424cd2d8531d"}
Nov 01 15:42:26.771542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.771501    5423 scope.go:122] "RemoveContainer" containerID="3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f"
Nov 01 15:42:26.781561 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.781541    5423 scope.go:122] "RemoveContainer" containerID="ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57"
Nov 01 15:42:26.790805 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.790777    5423 scope.go:122] "RemoveContainer" containerID="3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f"
Nov 01 15:42:26.791117 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:26.791092    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f\": container with ID starting with 3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f not found: ID does not exist" containerID="3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f"
Nov 01 15:42:26.791194 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.791124    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f"} err="failed to get container status \"3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f\": rpc error: code = NotFound desc = could not find container \"3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f\": container with ID starting with 3b5d536cc7fa7742509922e72b6c8876b345d5774f4b165fa71e04a40eef446f not found: ID does not exist"
Nov 01 15:42:26.791194 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.791143    5423 scope.go:122] "RemoveContainer" containerID="ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57"
Nov 01 15:42:26.791443 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:26.791415    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57\": container with ID starting with ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57 not found: ID does not exist" containerID="ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57"
Nov 01 15:42:26.791541 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.791451    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57"} err="failed to get container status \"ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57\": rpc error: code = NotFound desc = could not find container \"ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57\": container with ID starting with ec95b1ef0c3ea4790b77d0c798755bc1ec7cfe07e77322543127b9a6c2757f57 not found: ID does not exist"
Nov 01 15:42:26.937328 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.937250    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"]
Nov 01 15:42:26.942322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:26.942292    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-f4twj"]
Nov 01 15:42:27.961166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:27.960836    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" path="/var/lib/kubelet/pods/61f5dd4e-c04a-47ec-8291-5a6565e93bde/volumes"
Nov 01 15:42:28.891711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:28.891643    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" podUID="802ca007-2dfb-466e-bb41-a797ad8528b7" containerName="payload-processing" containerID="cri-o://9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac" gracePeriod=25
Nov 01 15:42:28.997965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:28.997922    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" podUID="7973261d-8cc2-4f1f-857f-c5d783a54908" containerName="payload-pre-processing" containerID="cri-o://6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd" gracePeriod=25
Nov 01 15:42:29.090120 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:29.090077    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" podUID="7973261d-8cc2-4f1f-857f-c5d783a54908" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:42:29.090311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:29.090084    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" podUID="802ca007-2dfb-466e-bb41-a797ad8528b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:42:30.675971 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:30.675933    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv" podUID="4c209a58-3ede-47f3-81dd-813a0ba66507" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:42:35.148792 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:35.148758    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:42:38.393705 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.393674    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:42:38.505562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.505534    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"]
Nov 01 15:42:38.505787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.505763    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f" podUID="5ecf7f83-bf59-427b-af68-07b70f1d7385" containerName="maas-api" containerID="cri-o://e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12" gracePeriod=30
Nov 01 15:42:38.768415 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.768379    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 15:42:38.836119 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.836087    5423 generic.go:356] "Generic (PLEG): container finished" podID="5ecf7f83-bf59-427b-af68-07b70f1d7385" containerID="e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12" exitCode=0
Nov 01 15:42:38.836283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.836121    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f" event={"ID":"5ecf7f83-bf59-427b-af68-07b70f1d7385","Type":"ContainerDied","Data":"e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12"}
Nov 01 15:42:38.836283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.836175    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f" event={"ID":"5ecf7f83-bf59-427b-af68-07b70f1d7385","Type":"ContainerDied","Data":"08667badfc4567079a6f6ee7238afbc076e14121dde040195f7d3ece7d1c720c"}
Nov 01 15:42:38.836283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.836196    5423 scope.go:122] "RemoveContainer" containerID="e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12"
Nov 01 15:42:38.836283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.836130    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"
Nov 01 15:42:38.844558 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.844534    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-metrics-certs\") pod \"5ecf7f83-bf59-427b-af68-07b70f1d7385\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") "
Nov 01 15:42:38.844678 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.844573    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ecf7f83-bf59-427b-af68-07b70f1d7385-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf7f83-bf59-427b-af68-07b70f1d7385-openshift-service-ca\") pod \"5ecf7f83-bf59-427b-af68-07b70f1d7385\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") "
Nov 01 15:42:38.844678 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.844624    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecf7f83-bf59-427b-af68-07b70f1d7385-kube-api-access-mvb69\" (UniqueName: \"kubernetes.io/projected/5ecf7f83-bf59-427b-af68-07b70f1d7385-kube-api-access-mvb69\") pod \"5ecf7f83-bf59-427b-af68-07b70f1d7385\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") "
Nov 01 15:42:38.844808 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.844737    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls\") pod \"5ecf7f83-bf59-427b-af68-07b70f1d7385\" (UID: \"5ecf7f83-bf59-427b-af68-07b70f1d7385\") "
Nov 01 15:42:38.844985 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.844952    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecf7f83-bf59-427b-af68-07b70f1d7385-openshift-service-ca" pod "5ecf7f83-bf59-427b-af68-07b70f1d7385" (UID: "5ecf7f83-bf59-427b-af68-07b70f1d7385"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:38.845171 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.845112    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ecf7f83-bf59-427b-af68-07b70f1d7385-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:38.845171 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.845157    5423 scope.go:122] "RemoveContainer" containerID="e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12"
Nov 01 15:42:38.845484 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:38.845459    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12\": container with ID starting with e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12 not found: ID does not exist" containerID="e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12"
Nov 01 15:42:38.845559 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.845495    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12"} err="failed to get container status \"e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12\": rpc error: code = NotFound desc = could not find container \"e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12\": container with ID starting with e0e388cf7d6b617e8ee7fca904d8dce7dbe173428f21a240a45b8dbef8442a12 not found: ID does not exist"
Nov 01 15:42:38.846779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.846752    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecf7f83-bf59-427b-af68-07b70f1d7385-kube-api-access-mvb69" pod "5ecf7f83-bf59-427b-af68-07b70f1d7385" (UID: "5ecf7f83-bf59-427b-af68-07b70f1d7385"). InnerVolumeSpecName "kube-api-access-mvb69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:38.846779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.846748    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls" pod "5ecf7f83-bf59-427b-af68-07b70f1d7385" (UID: "5ecf7f83-bf59-427b-af68-07b70f1d7385"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:38.846897 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.846826    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-metrics-certs" pod "5ecf7f83-bf59-427b-af68-07b70f1d7385" (UID: "5ecf7f83-bf59-427b-af68-07b70f1d7385"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:38.946336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.946261    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvb69\" (UniqueName: \"kubernetes.io/projected/5ecf7f83-bf59-427b-af68-07b70f1d7385-kube-api-access-mvb69\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:38.946336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.946289    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:38.946336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:38.946300    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ecf7f83-bf59-427b-af68-07b70f1d7385-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:39.090746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:39.090705    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" podUID="802ca007-2dfb-466e-bb41-a797ad8528b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:42:39.090935 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:39.090705    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" podUID="7973261d-8cc2-4f1f-857f-c5d783a54908" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:42:39.257812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:39.257730    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"]
Nov 01 15:42:39.261203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:39.261176    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97c87c-65c86fcc9b-qcw6f"]
Nov 01 15:42:39.960289 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:39.960259    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ecf7f83-bf59-427b-af68-07b70f1d7385" path="/var/lib/kubelet/pods/5ecf7f83-bf59-427b-af68-07b70f1d7385/volumes"
Nov 01 15:42:40.676177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:40.676144    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv" podUID="4c209a58-3ede-47f3-81dd-813a0ba66507" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:42:43.184828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:43.184804    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:42:43.207572 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:43.207551    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:42:46.879349 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:46.879314    5423 generic.go:356] "Generic (PLEG): container finished" podID="4c209a58-3ede-47f3-81dd-813a0ba66507" containerID="612a176f3c39252dd6135fb0641512a4105dab27d31624ce5bde6c187d0f31ef" exitCode=0
Nov 01 15:42:46.879692 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:46.879378    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv" event={"ID":"4c209a58-3ede-47f3-81dd-813a0ba66507","Type":"ContainerDied","Data":"612a176f3c39252dd6135fb0641512a4105dab27d31624ce5bde6c187d0f31ef"}
Nov 01 15:42:46.900211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:46.900191    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 15:42:47.029252 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.029182    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c209a58-3ede-47f3-81dd-813a0ba66507-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c209a58-3ede-47f3-81dd-813a0ba66507-plugins-config-volume\") pod \"4c209a58-3ede-47f3-81dd-813a0ba66507\" (UID: \"4c209a58-3ede-47f3-81dd-813a0ba66507\") "
Nov 01 15:42:47.029444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.029308    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c209a58-3ede-47f3-81dd-813a0ba66507-kube-api-access-rkkqj\" (UniqueName: \"kubernetes.io/projected/4c209a58-3ede-47f3-81dd-813a0ba66507-kube-api-access-rkkqj\") pod \"4c209a58-3ede-47f3-81dd-813a0ba66507\" (UID: \"4c209a58-3ede-47f3-81dd-813a0ba66507\") "
Nov 01 15:42:47.029605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.029576    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c209a58-3ede-47f3-81dd-813a0ba66507-plugins-config-volume" pod "4c209a58-3ede-47f3-81dd-813a0ba66507" (UID: "4c209a58-3ede-47f3-81dd-813a0ba66507"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:47.031353 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.031328    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c209a58-3ede-47f3-81dd-813a0ba66507-kube-api-access-rkkqj" pod "4c209a58-3ede-47f3-81dd-813a0ba66507" (UID: "4c209a58-3ede-47f3-81dd-813a0ba66507"). InnerVolumeSpecName "kube-api-access-rkkqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:47.130679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.130654    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c209a58-3ede-47f3-81dd-813a0ba66507-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:47.130679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.130676    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkkqj\" (UniqueName: \"kubernetes.io/projected/4c209a58-3ede-47f3-81dd-813a0ba66507-kube-api-access-rkkqj\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:47.885060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.885029    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"
Nov 01 15:42:47.885060 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.885035    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv" event={"ID":"4c209a58-3ede-47f3-81dd-813a0ba66507","Type":"ContainerDied","Data":"b5ec046cb7302bd23d422c6093fe0db56465c1982eea6445412ae63c8e3ee9a1"}
Nov 01 15:42:47.885567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:47.885089    5423 scope.go:122] "RemoveContainer" containerID="612a176f3c39252dd6135fb0641512a4105dab27d31624ce5bde6c187d0f31ef"
Nov 01 15:42:48.050965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:48.050933    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"]
Nov 01 15:42:48.101808 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:48.101775    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97c87c-84bb9cd4fc-589wv"]
Nov 01 15:42:49.089133 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:42:49.089101    5423 logging.go:55] [core] [Channel #13996 SubChannel #13998]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.180:50052", ServerName: "10.133.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.180:50052: connect: connection refused"
Nov 01 15:42:49.142647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.142622    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:42:49.249430 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.249404    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802ca007-2dfb-466e-bb41-a797ad8528b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802ca007-2dfb-466e-bb41-a797ad8528b7-plugins-config-volume\") pod \"802ca007-2dfb-466e-bb41-a797ad8528b7\" (UID: \"802ca007-2dfb-466e-bb41-a797ad8528b7\") "
Nov 01 15:42:49.249583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.249438    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802ca007-2dfb-466e-bb41-a797ad8528b7-kube-api-access-5gvkh\" (UniqueName: \"kubernetes.io/projected/802ca007-2dfb-466e-bb41-a797ad8528b7-kube-api-access-5gvkh\") pod \"802ca007-2dfb-466e-bb41-a797ad8528b7\" (UID: \"802ca007-2dfb-466e-bb41-a797ad8528b7\") "
Nov 01 15:42:49.249758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.249738    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802ca007-2dfb-466e-bb41-a797ad8528b7-plugins-config-volume" pod "802ca007-2dfb-466e-bb41-a797ad8528b7" (UID: "802ca007-2dfb-466e-bb41-a797ad8528b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:49.250149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.250130    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:42:49.251767 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.251742    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802ca007-2dfb-466e-bb41-a797ad8528b7-kube-api-access-5gvkh" pod "802ca007-2dfb-466e-bb41-a797ad8528b7" (UID: "802ca007-2dfb-466e-bb41-a797ad8528b7"). InnerVolumeSpecName "kube-api-access-5gvkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:49.354473 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.354420    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7973261d-8cc2-4f1f-857f-c5d783a54908-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7973261d-8cc2-4f1f-857f-c5d783a54908-plugins-config-volume\") pod \"7973261d-8cc2-4f1f-857f-c5d783a54908\" (UID: \"7973261d-8cc2-4f1f-857f-c5d783a54908\") "
Nov 01 15:42:49.354788 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.354764    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7973261d-8cc2-4f1f-857f-c5d783a54908-kube-api-access-m27ns\" (UniqueName: \"kubernetes.io/projected/7973261d-8cc2-4f1f-857f-c5d783a54908-kube-api-access-m27ns\") pod \"7973261d-8cc2-4f1f-857f-c5d783a54908\" (UID: \"7973261d-8cc2-4f1f-857f-c5d783a54908\") "
Nov 01 15:42:49.355207 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.355163    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802ca007-2dfb-466e-bb41-a797ad8528b7-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:49.359088 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.355190    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gvkh\" (UniqueName: \"kubernetes.io/projected/802ca007-2dfb-466e-bb41-a797ad8528b7-kube-api-access-5gvkh\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:49.359088 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.355588    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7973261d-8cc2-4f1f-857f-c5d783a54908-plugins-config-volume" pod "7973261d-8cc2-4f1f-857f-c5d783a54908" (UID: "7973261d-8cc2-4f1f-857f-c5d783a54908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:49.359231 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.357194    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7973261d-8cc2-4f1f-857f-c5d783a54908-kube-api-access-m27ns" pod "7973261d-8cc2-4f1f-857f-c5d783a54908" (UID: "7973261d-8cc2-4f1f-857f-c5d783a54908"). InnerVolumeSpecName "kube-api-access-m27ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:49.460110 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.460029    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7973261d-8cc2-4f1f-857f-c5d783a54908-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:49.460110 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.460062    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m27ns\" (UniqueName: \"kubernetes.io/projected/7973261d-8cc2-4f1f-857f-c5d783a54908-kube-api-access-m27ns\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:49.898497 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.898462    5423 generic.go:356] "Generic (PLEG): container finished" podID="7973261d-8cc2-4f1f-857f-c5d783a54908" containerID="6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd" exitCode=0
Nov 01 15:42:49.898699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.898525    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:42:49.898699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.898547    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" event={"ID":"7973261d-8cc2-4f1f-857f-c5d783a54908","Type":"ContainerDied","Data":"6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd"}
Nov 01 15:42:49.898699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.898643    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" event={"ID":"7973261d-8cc2-4f1f-857f-c5d783a54908","Type":"ContainerDied","Data":"921a10d3a76c78a79b97ea299d7e159f40e876bf21f4735f0b73380796262c10"}
Nov 01 15:42:49.898699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.898673    5423 scope.go:122] "RemoveContainer" containerID="6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd"
Nov 01 15:42:49.900214 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.900193    5423 generic.go:356] "Generic (PLEG): container finished" podID="802ca007-2dfb-466e-bb41-a797ad8528b7" containerID="9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac" exitCode=0
Nov 01 15:42:49.900352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.900245    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" event={"ID":"802ca007-2dfb-466e-bb41-a797ad8528b7","Type":"ContainerDied","Data":"9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac"}
Nov 01 15:42:49.900352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.900274    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:42:49.900352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.900278    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" event={"ID":"802ca007-2dfb-466e-bb41-a797ad8528b7","Type":"ContainerDied","Data":"950def2a9f29a4befc68828545682d826290861790080a1346d7bf25711bbd62"}
Nov 01 15:42:49.917523 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.917507    5423 scope.go:122] "RemoveContainer" containerID="6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd"
Nov 01 15:42:49.917754 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:49.917739    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd\": container with ID starting with 6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd not found: ID does not exist" containerID="6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd"
Nov 01 15:42:49.917796 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.917761    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd"} err="failed to get container status \"6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd\": rpc error: code = NotFound desc = could not find container \"6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd\": container with ID starting with 6454095150ff68338662cdfb6aac59220241329abd2d4ae65ef761c6149b58fd not found: ID does not exist"
Nov 01 15:42:49.917796 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.917776    5423 scope.go:122] "RemoveContainer" containerID="9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac"
Nov 01 15:42:49.925501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.925479    5423 scope.go:122] "RemoveContainer" containerID="9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac"
Nov 01 15:42:49.925758 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:42:49.925738    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac\": container with ID starting with 9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac not found: ID does not exist" containerID="9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac"
Nov 01 15:42:49.925820 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.925763    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac"} err="failed to get container status \"9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac\": rpc error: code = NotFound desc = could not find container \"9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac\": container with ID starting with 9806e4171ee7c97cb774bcf5c14b51ecc472368bf35b3d4f4e8765abda7402ac not found: ID does not exist"
Nov 01 15:42:49.960697 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:49.960676    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c209a58-3ede-47f3-81dd-813a0ba66507" path="/var/lib/kubelet/pods/4c209a58-3ede-47f3-81dd-813a0ba66507/volumes"
Nov 01 15:42:50.064650 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:50.064619    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"]
Nov 01 15:42:50.089560 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:50.089528    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59" podUID="802ca007-2dfb-466e-bb41-a797ad8528b7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.181:50052\" within 1s: context deadline exceeded"
Nov 01 15:42:50.089940 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:42:50.089603    5423 logging.go:55] [core] [Channel #13995 SubChannel #13997]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.181:50052", ServerName: "10.133.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.181:50052: operation was canceled"
Nov 01 15:42:50.089940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:50.089610    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"
Nov 01 15:42:50.089940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:50.089634    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh" podUID="7973261d-8cc2-4f1f-857f-c5d783a54908" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.180:50052\" within 1s: context deadline exceeded"
Nov 01 15:42:50.089940 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:42:50.089673    5423 logging.go:55] [core] [Channel #13996 SubChannel #13998]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.180:50052", ServerName: "10.133.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.180:50052: operation was canceled"
Nov 01 15:42:50.089940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:50.089719    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"
Nov 01 15:42:50.116509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:50.116488    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ac11c-566485cc99-blq59"]
Nov 01 15:42:50.225607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:50.225536    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"]
Nov 01 15:42:50.230112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:50.230085    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ac11c-57b8bfb4cf-fsrxh"]
Nov 01 15:42:51.960277 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:51.960232    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7973261d-8cc2-4f1f-857f-c5d783a54908" path="/var/lib/kubelet/pods/7973261d-8cc2-4f1f-857f-c5d783a54908/volumes"
Nov 01 15:42:51.960675 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:51.960644    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802ca007-2dfb-466e-bb41-a797ad8528b7" path="/var/lib/kubelet/pods/802ca007-2dfb-466e-bb41-a797ad8528b7/volumes"
Nov 01 15:42:55.262043 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:55.262002    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:42:56.089699 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:56.089662    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"]
Nov 01 15:42:56.196156 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:42:56.196122    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"]
Nov 01 15:43:01.090966 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:01.090920    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2" podUID="fabf9cde-981f-4801-be3e-7ba67fa35f77" containerName="payload-processing" containerID="cri-o://adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53" gracePeriod=25
Nov 01 15:43:01.196785 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:01.196748    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm" podUID="f364e0c0-495f-4818-b6e5-e336fbab1ab6" containerName="payload-pre-processing" containerID="cri-o://f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936" gracePeriod=25
Nov 01 15:43:04.804111 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:04.804077    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm" podUID="f364e0c0-495f-4818-b6e5-e336fbab1ab6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:43:05.148625 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:05.148586    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:43:06.808375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:06.808338    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2" podUID="fabf9cde-981f-4801-be3e-7ba67fa35f77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:43:14.803953 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:14.803908    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm" podUID="f364e0c0-495f-4818-b6e5-e336fbab1ab6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:43:16.464051 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:16.464011    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"]
Nov 01 15:43:16.513775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:16.513736    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"]
Nov 01 15:43:16.807971 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:16.807934    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2" podUID="fabf9cde-981f-4801-be3e-7ba67fa35f77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:43:17.771003 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:17.770968    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:43:19.018931 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:19.018898    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"]
Nov 01 15:43:19.019364 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:19.019164    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy" containerID="cri-o://368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1" gracePeriod=30
Nov 01 15:43:21.340810 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.340781    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:43:21.384048 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.384012    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabf9cde-981f-4801-be3e-7ba67fa35f77-kube-api-access-2bgfq\" (UniqueName: \"kubernetes.io/projected/fabf9cde-981f-4801-be3e-7ba67fa35f77-kube-api-access-2bgfq\") pod \"fabf9cde-981f-4801-be3e-7ba67fa35f77\" (UID: \"fabf9cde-981f-4801-be3e-7ba67fa35f77\") "
Nov 01 15:43:21.384222 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.384201    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fabf9cde-981f-4801-be3e-7ba67fa35f77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fabf9cde-981f-4801-be3e-7ba67fa35f77-plugins-config-volume\") pod \"fabf9cde-981f-4801-be3e-7ba67fa35f77\" (UID: \"fabf9cde-981f-4801-be3e-7ba67fa35f77\") "
Nov 01 15:43:21.384569 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.384544    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabf9cde-981f-4801-be3e-7ba67fa35f77-plugins-config-volume" pod "fabf9cde-981f-4801-be3e-7ba67fa35f77" (UID: "fabf9cde-981f-4801-be3e-7ba67fa35f77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:43:21.386150 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.386127    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabf9cde-981f-4801-be3e-7ba67fa35f77-kube-api-access-2bgfq" pod "fabf9cde-981f-4801-be3e-7ba67fa35f77" (UID: "fabf9cde-981f-4801-be3e-7ba67fa35f77"). InnerVolumeSpecName "kube-api-access-2bgfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:21.435901 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.435875    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:43:21.485287 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.485257    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f364e0c0-495f-4818-b6e5-e336fbab1ab6-kube-api-access-jhcnn\" (UniqueName: \"kubernetes.io/projected/f364e0c0-495f-4818-b6e5-e336fbab1ab6-kube-api-access-jhcnn\") pod \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\" (UID: \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\") "
Nov 01 15:43:21.485434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.485356    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f364e0c0-495f-4818-b6e5-e336fbab1ab6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f364e0c0-495f-4818-b6e5-e336fbab1ab6-plugins-config-volume\") pod \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\" (UID: \"f364e0c0-495f-4818-b6e5-e336fbab1ab6\") "
Nov 01 15:43:21.485680 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.485655    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f364e0c0-495f-4818-b6e5-e336fbab1ab6-plugins-config-volume" pod "f364e0c0-495f-4818-b6e5-e336fbab1ab6" (UID: "f364e0c0-495f-4818-b6e5-e336fbab1ab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:43:21.485794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.485733    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fabf9cde-981f-4801-be3e-7ba67fa35f77-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:21.485794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.485748    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bgfq\" (UniqueName: \"kubernetes.io/projected/fabf9cde-981f-4801-be3e-7ba67fa35f77-kube-api-access-2bgfq\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:21.485794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.485757    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f364e0c0-495f-4818-b6e5-e336fbab1ab6-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:21.487246 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.487224    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f364e0c0-495f-4818-b6e5-e336fbab1ab6-kube-api-access-jhcnn" pod "f364e0c0-495f-4818-b6e5-e336fbab1ab6" (UID: "f364e0c0-495f-4818-b6e5-e336fbab1ab6"). InnerVolumeSpecName "kube-api-access-jhcnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:21.587754 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:21.587715    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhcnn\" (UniqueName: \"kubernetes.io/projected/f364e0c0-495f-4818-b6e5-e336fbab1ab6-kube-api-access-jhcnn\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:22.079446 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.079405    5423 generic.go:356] "Generic (PLEG): container finished" podID="f364e0c0-495f-4818-b6e5-e336fbab1ab6" containerID="f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936" exitCode=0
Nov 01 15:43:22.079639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.079460    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"
Nov 01 15:43:22.079639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.079541    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm" event={"ID":"f364e0c0-495f-4818-b6e5-e336fbab1ab6","Type":"ContainerDied","Data":"f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936"}
Nov 01 15:43:22.079639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.079595    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm" event={"ID":"f364e0c0-495f-4818-b6e5-e336fbab1ab6","Type":"ContainerDied","Data":"1e32f71e74f83956ec381ab828b806658eab00e9c0924714621314ecd2165784"}
Nov 01 15:43:22.079639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.079626    5423 scope.go:122] "RemoveContainer" containerID="f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936"
Nov 01 15:43:22.081044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.081023    5423 generic.go:356] "Generic (PLEG): container finished" podID="fabf9cde-981f-4801-be3e-7ba67fa35f77" containerID="adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53" exitCode=0
Nov 01 15:43:22.081177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.081073    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"
Nov 01 15:43:22.081177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.081081    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2" event={"ID":"fabf9cde-981f-4801-be3e-7ba67fa35f77","Type":"ContainerDied","Data":"adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53"}
Nov 01 15:43:22.081177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.081119    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2" event={"ID":"fabf9cde-981f-4801-be3e-7ba67fa35f77","Type":"ContainerDied","Data":"f3ae25c4f431914398fa01004a480ae336e1a17bd7ddb886e8ef28e15c14ce6b"}
Nov 01 15:43:22.088517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.088427    5423 scope.go:122] "RemoveContainer" containerID="f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936"
Nov 01 15:43:22.088688 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:22.088668    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936\": container with ID starting with f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936 not found: ID does not exist" containerID="f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936"
Nov 01 15:43:22.088733 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.088695    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936"} err="failed to get container status \"f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936\": rpc error: code = NotFound desc = could not find container \"f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936\": container with ID starting with f6eb1c54bfda57412959755478ff67e527343608981d8791254c2a6f85536936 not found: ID does not exist"
Nov 01 15:43:22.088733 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.088710    5423 scope.go:122] "RemoveContainer" containerID="adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53"
Nov 01 15:43:22.097317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.097299    5423 scope.go:122] "RemoveContainer" containerID="adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53"
Nov 01 15:43:22.097559 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:22.097541    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53\": container with ID starting with adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53 not found: ID does not exist" containerID="adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53"
Nov 01 15:43:22.097632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.097564    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53"} err="failed to get container status \"adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53\": rpc error: code = NotFound desc = could not find container \"adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53\": container with ID starting with adb96bf51723e767e5b816e893bd663e303e02e51c433ae02ff1ca0edada0f53 not found: ID does not exist"
Nov 01 15:43:22.246062 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.245989    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"]
Nov 01 15:43:22.296925 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.296891    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-52fffa-6d8c7f4569-sz5f2"]
Nov 01 15:43:22.361038 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.361004    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"]
Nov 01 15:43:22.368792 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:22.368757    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-52fffa-6bd5bf77bc-8skxm"]
Nov 01 15:43:23.959370 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:23.959334    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f364e0c0-495f-4818-b6e5-e336fbab1ab6" path="/var/lib/kubelet/pods/f364e0c0-495f-4818-b6e5-e336fbab1ab6/volumes"
Nov 01 15:43:23.959800 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:23.959784    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabf9cde-981f-4801-be3e-7ba67fa35f77" path="/var/lib/kubelet/pods/fabf9cde-981f-4801-be3e-7ba67fa35f77/volumes"
Nov 01 15:43:24.276678 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.276653    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:43:24.419894 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.419860    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-socket\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.419894 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.419900    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0966ad2c-b364-4404-9fdc-479744097176-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0966ad2c-b364-4404-9fdc-479744097176-istio-podinfo\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.420101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.419925    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-istio-token\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-istio-token\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.420101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.419947    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-credential-socket\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.420101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420038    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-certs\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.420256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420128    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-envoy\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.420256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420157    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-data\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.420256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420187    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-kube-api-access-qrwd2\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-kube-api-access-qrwd2\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.420256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420124    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-socket" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:24.420256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420225    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0966ad2c-b364-4404-9fdc-479744097176-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0966ad2c-b364-4404-9fdc-479744097176-istiod-ca-cert\") pod \"0966ad2c-b364-4404-9fdc-479744097176\" (UID: \"0966ad2c-b364-4404-9fdc-479744097176\") "
Nov 01 15:43:24.420571 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420339    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-certs" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:24.420571 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420356    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-credential-socket" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:24.420571 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420480    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-data" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:24.420780 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420731    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0966ad2c-b364-4404-9fdc-479744097176-istiod-ca-cert" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:43:24.420780 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420734    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:24.420861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420779    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:24.420861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420796    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:24.420861 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.420814    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:24.422409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.422308    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-envoy" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:24.422409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.422358    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-istio-token" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:24.422611 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.422529    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0966ad2c-b364-4404-9fdc-479744097176-istio-podinfo" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:43:24.422949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.422926    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-kube-api-access-qrwd2" pod "0966ad2c-b364-4404-9fdc-479744097176" (UID: "0966ad2c-b364-4404-9fdc-479744097176"). InnerVolumeSpecName "kube-api-access-qrwd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:24.521886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.521806    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0966ad2c-b364-4404-9fdc-479744097176-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:24.521886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.521836    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwd2\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-kube-api-access-qrwd2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:24.521886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.521848    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0966ad2c-b364-4404-9fdc-479744097176-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:24.521886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.521856    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0966ad2c-b364-4404-9fdc-479744097176-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:24.521886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:24.521865    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0966ad2c-b364-4404-9fdc-479744097176-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:25.103091 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.103052    5423 generic.go:356] "Generic (PLEG): container finished" podID="0966ad2c-b364-4404-9fdc-479744097176" containerID="368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1" exitCode=0
Nov 01 15:43:25.103594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.103110    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"
Nov 01 15:43:25.103594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.103136    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" event={"ID":"0966ad2c-b364-4404-9fdc-479744097176","Type":"ContainerDied","Data":"368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1"}
Nov 01 15:43:25.103594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.103195    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr" event={"ID":"0966ad2c-b364-4404-9fdc-479744097176","Type":"ContainerDied","Data":"ba87847cee8edd54e267bb0d55990b8490a0925f60561741a6ce18701f40a071"}
Nov 01 15:43:25.103594 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.103222    5423 scope.go:122] "RemoveContainer" containerID="368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1"
Nov 01 15:43:25.113219 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.113201    5423 scope.go:122] "RemoveContainer" containerID="42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f"
Nov 01 15:43:25.123363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.123346    5423 scope.go:122] "RemoveContainer" containerID="368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1"
Nov 01 15:43:25.123715 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:25.123689    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1\": container with ID starting with 368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1 not found: ID does not exist" containerID="368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1"
Nov 01 15:43:25.123807 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.123721    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1"} err="failed to get container status \"368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1\": rpc error: code = NotFound desc = could not find container \"368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1\": container with ID starting with 368a14ca827fac5b9e7770bd58f13663d98f42bd4f9195cfe5a28dd9bd16f2c1 not found: ID does not exist"
Nov 01 15:43:25.123807 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.123747    5423 scope.go:122] "RemoveContainer" containerID="42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f"
Nov 01 15:43:25.124075 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:25.124043    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f\": container with ID starting with 42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f not found: ID does not exist" containerID="42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f"
Nov 01 15:43:25.124127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.124085    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f"} err="failed to get container status \"42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f\": rpc error: code = NotFound desc = could not find container \"42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f\": container with ID starting with 42c4bc5c82cf2ce90134420a1828843218fc0973bc4cb022625fab6929f1f20f not found: ID does not exist"
Nov 01 15:43:25.171819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.171790    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"]
Nov 01 15:43:25.269242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.269216    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"]
Nov 01 15:43:25.279511 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.279474    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-52fffa-openshift-default-5c9869d96c-6rjzr"]
Nov 01 15:43:25.962120 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:25.962077    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0966ad2c-b364-4404-9fdc-479744097176" path="/var/lib/kubelet/pods/0966ad2c-b364-4404-9fdc-479744097176/volumes"
Nov 01 15:43:30.172651 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:30.172608    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4" podUID="53047b0f-dbe2-4447-874d-7b6d47cbf3fa" containerName="payload-processing" containerID="cri-o://0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2" gracePeriod=25
Nov 01 15:43:30.421759 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:30.421732    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 15:43:30.592532 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:30.592502    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-kube-api-access-fljrk\" (UniqueName: \"kubernetes.io/projected/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-kube-api-access-fljrk\") pod \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\" (UID: \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\") "
Nov 01 15:43:30.592709 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:30.592551    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-plugins-config-volume\") pod \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\" (UID: \"53047b0f-dbe2-4447-874d-7b6d47cbf3fa\") "
Nov 01 15:43:30.592942 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:30.592919    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-plugins-config-volume" pod "53047b0f-dbe2-4447-874d-7b6d47cbf3fa" (UID: "53047b0f-dbe2-4447-874d-7b6d47cbf3fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:43:30.594608 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:30.594589    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-kube-api-access-fljrk" pod "53047b0f-dbe2-4447-874d-7b6d47cbf3fa" (UID: "53047b0f-dbe2-4447-874d-7b6d47cbf3fa"). InnerVolumeSpecName "kube-api-access-fljrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:30.693461 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:30.693426    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:30.693461 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:30.693454    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fljrk\" (UniqueName: \"kubernetes.io/projected/53047b0f-dbe2-4447-874d-7b6d47cbf3fa-kube-api-access-fljrk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:31.133700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.133661    5423 generic.go:356] "Generic (PLEG): container finished" podID="53047b0f-dbe2-4447-874d-7b6d47cbf3fa" containerID="0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2" exitCode=0
Nov 01 15:43:31.133881 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.133708    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"
Nov 01 15:43:31.133881 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.133744    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4" event={"ID":"53047b0f-dbe2-4447-874d-7b6d47cbf3fa","Type":"ContainerDied","Data":"0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2"}
Nov 01 15:43:31.133881 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.133803    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4" event={"ID":"53047b0f-dbe2-4447-874d-7b6d47cbf3fa","Type":"ContainerDied","Data":"c53e4c7b25ed998b27f61597b2125421912abd728405865cc3d7c6aeb59a0444"}
Nov 01 15:43:31.133881 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.133826    5423 scope.go:122] "RemoveContainer" containerID="0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2"
Nov 01 15:43:31.142750 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.142730    5423 scope.go:122] "RemoveContainer" containerID="0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2"
Nov 01 15:43:31.143000 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:31.142977    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2\": container with ID starting with 0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2 not found: ID does not exist" containerID="0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2"
Nov 01 15:43:31.143068 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.143009    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2"} err="failed to get container status \"0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2\": rpc error: code = NotFound desc = could not find container \"0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2\": container with ID starting with 0116921141acbab1c25fb39670d1fcf3a1ea2cb68193600b05eda533c56ccfa2 not found: ID does not exist"
Nov 01 15:43:31.299628 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.299592    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"]
Nov 01 15:43:31.350059 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.350027    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5b0dcc-7f67ff45f5-xx6c4"]
Nov 01 15:43:31.484725 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.484635    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" containerID="cri-o://d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8" gracePeriod=45
Nov 01 15:43:31.533296 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.533248    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" containerID="cri-o://5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656" gracePeriod=45
Nov 01 15:43:31.574954 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.574924    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Nov 01 15:43:31.632174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.632137    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:43:31.807269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.805496    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 15:43:31.828999 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.828976    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 15:43:31.907403 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.907362    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e19a5189-f91f-4561-9fa5-082efe591e41-tls-certs\" (UniqueName: \"kubernetes.io/secret/e19a5189-f91f-4561-9fa5-082efe591e41-tls-certs\") pod \"e19a5189-f91f-4561-9fa5-082efe591e41\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") "
Nov 01 15:43:31.907605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.907497    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-dshm\") pod \"e19a5189-f91f-4561-9fa5-082efe591e41\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") "
Nov 01 15:43:31.907605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.907521    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-tmp-dir\") pod \"e19a5189-f91f-4561-9fa5-082efe591e41\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") "
Nov 01 15:43:31.907605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.907547    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-home\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-home\") pod \"e19a5189-f91f-4561-9fa5-082efe591e41\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") "
Nov 01 15:43:31.907605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.907593    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-model-cache\") pod \"e19a5189-f91f-4561-9fa5-082efe591e41\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") "
Nov 01 15:43:31.907865 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.907632    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19a5189-f91f-4561-9fa5-082efe591e41-kube-api-access-j4xsk\" (UniqueName: \"kubernetes.io/projected/e19a5189-f91f-4561-9fa5-082efe591e41-kube-api-access-j4xsk\") pod \"e19a5189-f91f-4561-9fa5-082efe591e41\" (UID: \"e19a5189-f91f-4561-9fa5-082efe591e41\") "
Nov 01 15:43:31.907865 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.907809    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-tmp-dir" pod "e19a5189-f91f-4561-9fa5-082efe591e41" (UID: "e19a5189-f91f-4561-9fa5-082efe591e41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:31.908050 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.908005    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-model-cache" pod "e19a5189-f91f-4561-9fa5-082efe591e41" (UID: "e19a5189-f91f-4561-9fa5-082efe591e41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:31.908122 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.908112    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:31.908181 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.908131    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:31.908256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.908190    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-home" pod "e19a5189-f91f-4561-9fa5-082efe591e41" (UID: "e19a5189-f91f-4561-9fa5-082efe591e41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:31.909835 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.909813    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19a5189-f91f-4561-9fa5-082efe591e41-tls-certs" pod "e19a5189-f91f-4561-9fa5-082efe591e41" (UID: "e19a5189-f91f-4561-9fa5-082efe591e41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:31.909997 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.909979    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-dshm" pod "e19a5189-f91f-4561-9fa5-082efe591e41" (UID: "e19a5189-f91f-4561-9fa5-082efe591e41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:31.910073 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.909982    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19a5189-f91f-4561-9fa5-082efe591e41-kube-api-access-j4xsk" pod "e19a5189-f91f-4561-9fa5-082efe591e41" (UID: "e19a5189-f91f-4561-9fa5-082efe591e41"). InnerVolumeSpecName "kube-api-access-j4xsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:31.963205 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:31.963154    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53047b0f-dbe2-4447-874d-7b6d47cbf3fa" path="/var/lib/kubelet/pods/53047b0f-dbe2-4447-874d-7b6d47cbf3fa/volumes"
Nov 01 15:43:32.008562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.008481    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd534096-6aae-4990-b15c-bb5d262857a6-kube-api-access-lrgnl\" (UniqueName: \"kubernetes.io/projected/fd534096-6aae-4990-b15c-bb5d262857a6-kube-api-access-lrgnl\") pod \"fd534096-6aae-4990-b15c-bb5d262857a6\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") "
Nov 01 15:43:32.008562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.008541    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-dshm\") pod \"fd534096-6aae-4990-b15c-bb5d262857a6\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") "
Nov 01 15:43:32.008784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.008667    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-tmp-dir\") pod \"fd534096-6aae-4990-b15c-bb5d262857a6\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") "
Nov 01 15:43:32.008784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.008731    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-model-cache\") pod \"fd534096-6aae-4990-b15c-bb5d262857a6\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") "
Nov 01 15:43:32.008893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.008789    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-home\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-home\") pod \"fd534096-6aae-4990-b15c-bb5d262857a6\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") "
Nov 01 15:43:32.008893 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.008820    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd534096-6aae-4990-b15c-bb5d262857a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd534096-6aae-4990-b15c-bb5d262857a6-tls-certs\") pod \"fd534096-6aae-4990-b15c-bb5d262857a6\" (UID: \"fd534096-6aae-4990-b15c-bb5d262857a6\") "
Nov 01 15:43:32.009460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.009055    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-model-cache" pod "fd534096-6aae-4990-b15c-bb5d262857a6" (UID: "fd534096-6aae-4990-b15c-bb5d262857a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:32.009460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.009268    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.009460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.009285    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e19a5189-f91f-4561-9fa5-082efe591e41-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.009460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.009300    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xsk\" (UniqueName: \"kubernetes.io/projected/e19a5189-f91f-4561-9fa5-082efe591e41-kube-api-access-j4xsk\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.009460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.009315    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e19a5189-f91f-4561-9fa5-082efe591e41-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.009460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.009328    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.009460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.009362    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-home" pod "fd534096-6aae-4990-b15c-bb5d262857a6" (UID: "fd534096-6aae-4990-b15c-bb5d262857a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:32.009460 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.009437    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-tmp-dir" pod "fd534096-6aae-4990-b15c-bb5d262857a6" (UID: "fd534096-6aae-4990-b15c-bb5d262857a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:32.011208 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.011182    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-dshm" pod "fd534096-6aae-4990-b15c-bb5d262857a6" (UID: "fd534096-6aae-4990-b15c-bb5d262857a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:32.011420 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.011375    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd534096-6aae-4990-b15c-bb5d262857a6-tls-certs" pod "fd534096-6aae-4990-b15c-bb5d262857a6" (UID: "fd534096-6aae-4990-b15c-bb5d262857a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:32.011547 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.011490    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd534096-6aae-4990-b15c-bb5d262857a6-kube-api-access-lrgnl" pod "fd534096-6aae-4990-b15c-bb5d262857a6" (UID: "fd534096-6aae-4990-b15c-bb5d262857a6"). InnerVolumeSpecName "kube-api-access-lrgnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:32.110702 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.110661    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrgnl\" (UniqueName: \"kubernetes.io/projected/fd534096-6aae-4990-b15c-bb5d262857a6-kube-api-access-lrgnl\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.110702 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.110697    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.111269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.110712    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.111269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.110724    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd534096-6aae-4990-b15c-bb5d262857a6-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.111269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.110737    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd534096-6aae-4990-b15c-bb5d262857a6-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.140428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.140381    5423 generic.go:356] "Generic (PLEG): container finished" podID="e19a5189-f91f-4561-9fa5-082efe591e41" containerID="d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8" exitCode=0
Nov 01 15:43:32.140605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.140430    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" event={"ID":"e19a5189-f91f-4561-9fa5-082efe591e41","Type":"ContainerDied","Data":"d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8"}
Nov 01 15:43:32.140605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.140474    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt" event={"ID":"e19a5189-f91f-4561-9fa5-082efe591e41","Type":"ContainerDied","Data":"625c65e8f491eaf4812018609e1748d6ea433509452d318e221aa9cdde3f4b97"}
Nov 01 15:43:32.140605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.140438    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"
Nov 01 15:43:32.140605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.140493    5423 scope.go:122] "RemoveContainer" containerID="d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8"
Nov 01 15:43:32.142099 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.142078    5423 generic.go:356] "Generic (PLEG): container finished" podID="fd534096-6aae-4990-b15c-bb5d262857a6" containerID="5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656" exitCode=0
Nov 01 15:43:32.142240 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.142225    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"
Nov 01 15:43:32.142783 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.142755    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" event={"ID":"fd534096-6aae-4990-b15c-bb5d262857a6","Type":"ContainerDied","Data":"5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656"}
Nov 01 15:43:32.142964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.142853    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" event={"ID":"fd534096-6aae-4990-b15c-bb5d262857a6","Type":"ContainerDied","Data":"2f0bb24738b78c4bbc73c54c00e700925f448cf2b9f998dbb73d83a015ed2512"}
Nov 01 15:43:32.150084 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.150064    5423 scope.go:122] "RemoveContainer" containerID="d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8"
Nov 01 15:43:32.150339 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:32.150322    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8\": container with ID starting with d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8 not found: ID does not exist" containerID="d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8"
Nov 01 15:43:32.150413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.150347    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8"} err="failed to get container status \"d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8\": rpc error: code = NotFound desc = could not find container \"d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8\": container with ID starting with d1f8a70e3e043d42365742e416f287d237c6a19de55a880cab03ead4889ab7f8 not found: ID does not exist"
Nov 01 15:43:32.150413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.150365    5423 scope.go:122] "RemoveContainer" containerID="5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656"
Nov 01 15:43:32.160045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.160026    5423 scope.go:122] "RemoveContainer" containerID="5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656"
Nov 01 15:43:32.160343 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:32.160317    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656\": container with ID starting with 5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656 not found: ID does not exist" containerID="5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656"
Nov 01 15:43:32.160450 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.160344    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656"} err="failed to get container status \"5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656\": rpc error: code = NotFound desc = could not find container \"5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656\": container with ID starting with 5158d9408e4bfdfc8e3ee31db7a105d166eb9748cbbd47c9124645926483e656 not found: ID does not exist"
Nov 01 15:43:32.312442 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.312084    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"]
Nov 01 15:43:32.316602 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.316565    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf"]
Nov 01 15:43:32.430316 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.430267    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"]
Nov 01 15:43:32.434205 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.434178    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-2-w3-5b0dcc-kserve-b596d98d8-w96bt"]
Nov 01 15:43:32.779242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:32.779157    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/premium-w3-5b0dcc-kserve-7c6cc54b4d-sklpf" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:43:33.959141 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:33.959107    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" path="/var/lib/kubelet/pods/e19a5189-f91f-4561-9fa5-082efe591e41/volumes"
Nov 01 15:43:33.959598 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:33.959581    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" path="/var/lib/kubelet/pods/fd534096-6aae-4990-b15c-bb5d262857a6/volumes"
Nov 01 15:43:35.150731 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:35.150687    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:43:37.171343 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.171309    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"]
Nov 01 15:43:37.171843 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.171534    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c" podUID="610938fd-7cd7-475c-b75a-c095c335fc7a" containerName="maas-api" containerID="cri-o://a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a" gracePeriod=30
Nov 01 15:43:37.424472 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.424408    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 15:43:37.468100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.468069    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610938fd-7cd7-475c-b75a-c095c335fc7a-kube-api-access-rxf95\" (UniqueName: \"kubernetes.io/projected/610938fd-7cd7-475c-b75a-c095c335fc7a-kube-api-access-rxf95\") pod \"610938fd-7cd7-475c-b75a-c095c335fc7a\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") "
Nov 01 15:43:37.468100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.468101    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls\") pod \"610938fd-7cd7-475c-b75a-c095c335fc7a\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") "
Nov 01 15:43:37.468356 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.468122    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610938fd-7cd7-475c-b75a-c095c335fc7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610938fd-7cd7-475c-b75a-c095c335fc7a-openshift-service-ca\") pod \"610938fd-7cd7-475c-b75a-c095c335fc7a\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") "
Nov 01 15:43:37.468356 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.468158    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-metrics-certs\") pod \"610938fd-7cd7-475c-b75a-c095c335fc7a\" (UID: \"610938fd-7cd7-475c-b75a-c095c335fc7a\") "
Nov 01 15:43:37.468587 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.468562    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610938fd-7cd7-475c-b75a-c095c335fc7a-openshift-service-ca" pod "610938fd-7cd7-475c-b75a-c095c335fc7a" (UID: "610938fd-7cd7-475c-b75a-c095c335fc7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:43:37.470341 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.470311    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-metrics-certs" pod "610938fd-7cd7-475c-b75a-c095c335fc7a" (UID: "610938fd-7cd7-475c-b75a-c095c335fc7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:37.470469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.470355    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls" pod "610938fd-7cd7-475c-b75a-c095c335fc7a" (UID: "610938fd-7cd7-475c-b75a-c095c335fc7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:37.470469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.470369    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610938fd-7cd7-475c-b75a-c095c335fc7a-kube-api-access-rxf95" pod "610938fd-7cd7-475c-b75a-c095c335fc7a" (UID: "610938fd-7cd7-475c-b75a-c095c335fc7a"). InnerVolumeSpecName "kube-api-access-rxf95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:37.569815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.569773    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:37.569815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.569802    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxf95\" (UniqueName: \"kubernetes.io/projected/610938fd-7cd7-475c-b75a-c095c335fc7a-kube-api-access-rxf95\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:37.569815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.569814    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610938fd-7cd7-475c-b75a-c095c335fc7a-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:37.569815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:37.569822    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610938fd-7cd7-475c-b75a-c095c335fc7a-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:38.179779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.179747    5423 generic.go:356] "Generic (PLEG): container finished" podID="610938fd-7cd7-475c-b75a-c095c335fc7a" containerID="a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a" exitCode=0
Nov 01 15:43:38.180256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.179790    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c" event={"ID":"610938fd-7cd7-475c-b75a-c095c335fc7a","Type":"ContainerDied","Data":"a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a"}
Nov 01 15:43:38.180256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.179794    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"
Nov 01 15:43:38.180256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.179830    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c" event={"ID":"610938fd-7cd7-475c-b75a-c095c335fc7a","Type":"ContainerDied","Data":"0b39aee01e6828daff51429d89977e6d21b2e0cd51a7ab764a2ad7882da96550"}
Nov 01 15:43:38.180256 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.179853    5423 scope.go:122] "RemoveContainer" containerID="a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a"
Nov 01 15:43:38.188935 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.188916    5423 scope.go:122] "RemoveContainer" containerID="a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a"
Nov 01 15:43:38.189205 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:43:38.189182    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a\": container with ID starting with a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a not found: ID does not exist" containerID="a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a"
Nov 01 15:43:38.189275 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.189212    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a"} err="failed to get container status \"a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a\": rpc error: code = NotFound desc = could not find container \"a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a\": container with ID starting with a968a2bc7e9287ebd2a2d4b56451eb7aaaf44f8738e1d38f061095ece8e7193a not found: ID does not exist"
Nov 01 15:43:38.347029 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.346990    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"]
Nov 01 15:43:38.350169 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:38.350145    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5b0dcc-76956576ff-zhb5c"]
Nov 01 15:43:39.959628 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:39.959589    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610938fd-7cd7-475c-b75a-c095c335fc7a" path="/var/lib/kubelet/pods/610938fd-7cd7-475c-b75a-c095c335fc7a/volumes"
Nov 01 15:43:56.293096 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:56.293064    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"]
Nov 01 15:43:56.293606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:56.293272    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy" containerID="cri-o://7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a" gracePeriod=30
Nov 01 15:43:57.750021 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:57.749988    5423 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:43:57.750415 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:57.750037    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:43:59.357305 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:43:59.357273    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:44:01.553723 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.553696    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 15:44:01.637320 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637292    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-envoy\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637466 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637327    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-data\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637466 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637411    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-token\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-token\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637466 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637445    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-credential-socket\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637468    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-kube-api-access-cpcxb\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-kube-api-access-cpcxb\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637490    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-podinfo\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637527    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-certs\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637623 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637600    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-data" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.637831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637650    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d59485-1675-4de0-a87d-35fac5a4e730-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d59485-1675-4de0-a87d-35fac5a4e730-istiod-ca-cert\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637698    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-socket\") pod \"f0d59485-1675-4de0-a87d-35fac5a4e730\" (UID: \"f0d59485-1675-4de0-a87d-35fac5a4e730\") "
Nov 01 15:44:01.637831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.637748    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-credential-socket" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.638095 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.638073    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-socket" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.638175 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.638109    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d59485-1675-4de0-a87d-35fac5a4e730-istiod-ca-cert" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:44:01.638175 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.638168    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-certs" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.638286 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.638180    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.638286 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.638201    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.638286 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.638218    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.639583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.639559    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-envoy" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.639752 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.639729    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-podinfo" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:44:01.639853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.639830    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-kube-api-access-cpcxb" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "kube-api-access-cpcxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:44:01.639903 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.639876    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-token" pod "f0d59485-1675-4de0-a87d-35fac5a4e730" (UID: "f0d59485-1675-4de0-a87d-35fac5a4e730"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:44:01.739360 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.739321    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.739360 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.739350    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d59485-1675-4de0-a87d-35fac5a4e730-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.739360 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.739359    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.739360 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.739368    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.739635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.739378    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpcxb\" (UniqueName: \"kubernetes.io/projected/f0d59485-1675-4de0-a87d-35fac5a4e730-kube-api-access-cpcxb\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.739635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:01.739405    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d59485-1675-4de0-a87d-35fac5a4e730-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:02.298657 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.298577    5423 generic.go:356] "Generic (PLEG): container finished" podID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerID="7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a" exitCode=0
Nov 01 15:44:02.298657 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.298627    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"
Nov 01 15:44:02.298844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.298656    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" event={"ID":"f0d59485-1675-4de0-a87d-35fac5a4e730","Type":"ContainerDied","Data":"7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a"}
Nov 01 15:44:02.298844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.298709    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7" event={"ID":"f0d59485-1675-4de0-a87d-35fac5a4e730","Type":"ContainerDied","Data":"79d279b92712f03a20e38334c14062f36ee9357a04904dfdf83c43d333c4133c"}
Nov 01 15:44:02.298844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.298730    5423 scope.go:122] "RemoveContainer" containerID="7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a"
Nov 01 15:44:02.307175 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.307158    5423 scope.go:122] "RemoveContainer" containerID="66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950"
Nov 01 15:44:02.314712 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.314692    5423 scope.go:122] "RemoveContainer" containerID="7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a"
Nov 01 15:44:02.314977 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:44:02.314959    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a\": container with ID starting with 7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a not found: ID does not exist" containerID="7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a"
Nov 01 15:44:02.315056 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.314984    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a"} err="failed to get container status \"7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a\": rpc error: code = NotFound desc = could not find container \"7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a\": container with ID starting with 7710f95ff36a3c07fe144a4e4bab16fa0745b651a3d073b636a6a30a94f65f2a not found: ID does not exist"
Nov 01 15:44:02.315056 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.315000    5423 scope.go:122] "RemoveContainer" containerID="66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950"
Nov 01 15:44:02.315258 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:44:02.315239    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950\": container with ID starting with 66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950 not found: ID does not exist" containerID="66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950"
Nov 01 15:44:02.315300 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.315263    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950"} err="failed to get container status \"66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950\": rpc error: code = NotFound desc = could not find container \"66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950\": container with ID starting with 66ad87755faab50550d64809a37eeccf5f8ab4ff89741c8fbd6301a793312950 not found: ID does not exist"
Nov 01 15:44:02.516554 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.516524    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"]
Nov 01 15:44:02.670019 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:02.669989    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-rpws7"]
Nov 01 15:44:03.734668 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:03.734640    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"]
Nov 01 15:44:03.735216 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:03.734840    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" containerID="cri-o://2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712" gracePeriod=30
Nov 01 15:44:03.959919 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:03.959882    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" path="/var/lib/kubelet/pods/f0d59485-1675-4de0-a87d-35fac5a4e730/volumes"
Nov 01 15:44:04.801665 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:04.801631    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d985f759b-hcptd_590e6cf3-e96c-4b31-a570-555017c2bb47/authorino/0.log"
Nov 01 15:44:05.010678 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:05.010643    5423 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:44:05.010844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:05.010705    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:44:05.147255 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:44:05.147216    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:44:05.883330 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:05.883303    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-55crp_f521cd06-308b-48bb-a9a3-1ee4a82fbe04/kuadrant-console-plugin/0.log"
Nov 01 15:44:06.882265 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:06.882238    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dkfnx_a4d27b75-03ea-410d-b323-86daa17409e9/limitador/0.log"
Nov 01 15:44:08.149407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:08.149363    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9f778ddb6-jnqbk_0afcba4a-013c-42bd-8216-b9b91a2c4bde/manager/0.log"
Nov 01 15:44:08.979180 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:08.979157    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 15:44:09.108542 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108511    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-data\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.108745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108574    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-kube-api-access-qrtf2\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-kube-api-access-qrtf2\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.108745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108604    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-credential-socket\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.108745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108633    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-token\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-token\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.108745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108724    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istiod-ca-cert\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.108969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108778    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-socket\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.108969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108811    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-data" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:09.108969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108823    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-certs\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.108969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108892    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-podinfo\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.108969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.108924    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-envoy\") pod \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\" (UID: \"fa8fa78c-ccfb-4655-8245-2b7e74bfda44\") "
Nov 01 15:44:09.109216 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109048    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-certs" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:09.109216 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109171    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-credential-socket" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:09.109216 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109185    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-socket" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:09.109382 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109262    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istiod-ca-cert" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:44:09.109676 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109622    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.109676 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109649    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.109676 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109665    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.109676 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109677    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.110040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.109689    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.112040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.112017    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-envoy" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:09.112158 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.112093    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-token" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:44:09.112230 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.112206    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-kube-api-access-qrtf2" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "kube-api-access-qrtf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:44:09.112592 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.112561    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-podinfo" pod "fa8fa78c-ccfb-4655-8245-2b7e74bfda44" (UID: "fa8fa78c-ccfb-4655-8245-2b7e74bfda44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:44:09.210863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.210828    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.210863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.210856    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.210863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.210870    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrtf2\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-kube-api-access-qrtf2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.211332 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.210882    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8fa78c-ccfb-4655-8245-2b7e74bfda44-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:09.331997 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.331958    5423 generic.go:356] "Generic (PLEG): container finished" podID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerID="2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712" exitCode=0
Nov 01 15:44:09.332166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.331999    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"
Nov 01 15:44:09.332166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.332038    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" event={"ID":"fa8fa78c-ccfb-4655-8245-2b7e74bfda44","Type":"ContainerDied","Data":"2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712"}
Nov 01 15:44:09.332166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.332097    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r" event={"ID":"fa8fa78c-ccfb-4655-8245-2b7e74bfda44","Type":"ContainerDied","Data":"d457b317ec28f9eeb22f2ea50692122089532c42af017515f5ef2c4f70230aa5"}
Nov 01 15:44:09.332166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.332125    5423 scope.go:122] "RemoveContainer" containerID="2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712"
Nov 01 15:44:09.342096 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.342080    5423 scope.go:122] "RemoveContainer" containerID="2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712"
Nov 01 15:44:09.342380 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:44:09.342359    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712\": container with ID starting with 2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712 not found: ID does not exist" containerID="2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712"
Nov 01 15:44:09.342474 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.342402    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712"} err="failed to get container status \"2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712\": rpc error: code = NotFound desc = could not find container \"2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712\": container with ID starting with 2fd7b1a1466eb1387689f856e5f7b758ce0b847dfff2c47728a6c987e192f712 not found: ID does not exist"
Nov 01 15:44:09.549745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.549673    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"]
Nov 01 15:44:09.652606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.652580    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-wtw7r"]
Nov 01 15:44:09.776046 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.776014    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75bf7b5bf6-x48jt_1e87bc8c-e77e-42fe-ba31-b75c4a5967cc/manager/1.log"
Nov 01 15:44:09.958486 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:09.958450    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" path="/var/lib/kubelet/pods/fa8fa78c-ccfb-4655-8245-2b7e74bfda44/volumes"
Nov 01 15:44:11.217592 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:11.217557    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75bf7b5bf6-x48jt_1e87bc8c-e77e-42fe-ba31-b75c4a5967cc/manager/0.log"
Nov 01 15:44:24.649762 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.649721    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"]
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650107    5423 cpu_manager.go:414] "containerMap: removing container" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650120    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650131    5423 cpu_manager.go:414] "containerMap: removing container" podUID="727d2dcc-9e55-4a28-8f6b-85510fb014fd" containerName="payload-processing"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650136    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727d2dcc-9e55-4a28-8f6b-85510fb014fd" containerName="payload-processing"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650143    5423 cpu_manager.go:414] "containerMap: removing container" podUID="05eced9e-3508-482d-9f58-90f9e0e16546" containerName="main"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650148    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05eced9e-3508-482d-9f58-90f9e0e16546" containerName="main"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650159    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650163    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650171    5423 cpu_manager.go:414] "containerMap: removing container" podUID="53047b0f-dbe2-4447-874d-7b6d47cbf3fa" containerName="payload-processing"
Nov 01 15:44:24.650174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650176    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53047b0f-dbe2-4447-874d-7b6d47cbf3fa" containerName="payload-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650181    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" containerName="payload-pre-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650186    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" containerName="payload-pre-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650193    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650198    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650203    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" containerName="payload-pre-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650207    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" containerName="payload-pre-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650215    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fe198337-d3c3-4aed-a9d3-8457fcfed95c" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650220    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe198337-d3c3-4aed-a9d3-8457fcfed95c" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650228    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fabf9cde-981f-4801-be3e-7ba67fa35f77" containerName="payload-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650233    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabf9cde-981f-4801-be3e-7ba67fa35f77" containerName="payload-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650238    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650243    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650251    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650258    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650274    5423 cpu_manager.go:414] "containerMap: removing container" podUID="31998e3a-cad5-4d68-8514-0daa9a09f09b" containerName="test-curl-7384-542e7e"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650280    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31998e3a-cad5-4d68-8514-0daa9a09f09b" containerName="test-curl-7384-542e7e"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650290    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f364e0c0-495f-4818-b6e5-e336fbab1ab6" containerName="payload-pre-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650297    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f364e0c0-495f-4818-b6e5-e336fbab1ab6" containerName="payload-pre-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650306    5423 cpu_manager.go:414] "containerMap: removing container" podUID="7973261d-8cc2-4f1f-857f-c5d783a54908" containerName="payload-pre-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650312    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7973261d-8cc2-4f1f-857f-c5d783a54908" containerName="payload-pre-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650324    5423 cpu_manager.go:414] "containerMap: removing container" podUID="610938fd-7cd7-475c-b75a-c095c335fc7a" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650331    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610938fd-7cd7-475c-b75a-c095c335fc7a" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650342    5423 cpu_manager.go:414] "containerMap: removing container" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650348    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650362    5423 cpu_manager.go:414] "containerMap: removing container" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650369    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650382    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650402    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650412    5423 cpu_manager.go:414] "containerMap: removing container" podUID="5ecf7f83-bf59-427b-af68-07b70f1d7385" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650419    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecf7f83-bf59-427b-af68-07b70f1d7385" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650428    5423 cpu_manager.go:414] "containerMap: removing container" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650434    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650445    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650451    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650460    5423 cpu_manager.go:414] "containerMap: removing container" podUID="354e828e-9394-4e85-894b-f2930448c67d" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650467    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354e828e-9394-4e85-894b-f2930448c67d" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650483    5423 cpu_manager.go:414] "containerMap: removing container" podUID="081a7f84-b0fc-46da-afbd-9b58fe9c6322" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650491    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081a7f84-b0fc-46da-afbd-9b58fe9c6322" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650503    5423 cpu_manager.go:414] "containerMap: removing container" podUID="802ca007-2dfb-466e-bb41-a797ad8528b7" containerName="payload-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650510    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802ca007-2dfb-466e-bb41-a797ad8528b7" containerName="payload-processing"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650522    5423 cpu_manager.go:414] "containerMap: removing container" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650528    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650537    5423 cpu_manager.go:414] "containerMap: removing container" podUID="bd1dec20-0f8a-4623-8f6f-683fd454ede4" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650543    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1dec20-0f8a-4623-8f6f-683fd454ede4" containerName="maas-api"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650552    5423 cpu_manager.go:414] "containerMap: removing container" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650558    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650569    5423 cpu_manager.go:414] "containerMap: removing container" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650575    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650584    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650589    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650597    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650602    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy"
Nov 01 15:44:24.650577 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650608    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4c209a58-3ede-47f3-81dd-813a0ba66507" containerName="payload-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650612    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c209a58-3ede-47f3-81dd-813a0ba66507" containerName="payload-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650690    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="53047b0f-dbe2-4447-874d-7b6d47cbf3fa" containerName="payload-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650696    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a0e0fa-f4e9-49d3-abcf-67c6cd089143" containerName="payload-pre-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650705    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650712    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2ae3bd-5205-41e3-a0b1-8c2fc1b4dfe3" containerName="payload-pre-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650718    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe198337-d3c3-4aed-a9d3-8457fcfed95c" containerName="maas-api"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650724    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabf9cde-981f-4801-be3e-7ba67fa35f77" containerName="payload-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650730    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650735    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74a9078-63f9-4173-bbe5-e893254c0f8d" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650741    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="31998e3a-cad5-4d68-8514-0daa9a09f09b" containerName="test-curl-7384-542e7e"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650747    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f364e0c0-495f-4818-b6e5-e336fbab1ab6" containerName="payload-pre-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650754    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="7973261d-8cc2-4f1f-857f-c5d783a54908" containerName="payload-pre-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650759    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="610938fd-7cd7-475c-b75a-c095c335fc7a" containerName="maas-api"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650764    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e30a1b-095a-450b-98af-ecae621e1047" containerName="main"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650769    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650774    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650779    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650784    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650789    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecf7f83-bf59-427b-af68-07b70f1d7385" containerName="maas-api"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650794    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650798    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d59485-1675-4de0-a87d-35fac5a4e730" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650804    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="354e828e-9394-4e85-894b-f2930448c67d" containerName="maas-api"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650810    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="081a7f84-b0fc-46da-afbd-9b58fe9c6322" containerName="maas-api"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650816    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="802ca007-2dfb-466e-bb41-a797ad8528b7" containerName="payload-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650822    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1dec20-0f8a-4623-8f6f-683fd454ede4" containerName="maas-api"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650827    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a367b54-8394-4670-9a95-0ad8e9e08c0c" containerName="main"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650831    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="29709b9a-d3f9-4795-a58a-9b2ba35950fb" containerName="main"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650836    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd534096-6aae-4990-b15c-bb5d262857a6" containerName="main"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650841    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8fa78c-ccfb-4655-8245-2b7e74bfda44" containerName="istio-proxy"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650846    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c209a58-3ede-47f3-81dd-813a0ba66507" containerName="payload-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650852    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="727d2dcc-9e55-4a28-8f6b-85510fb014fd" containerName="payload-processing"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650856    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="05eced9e-3508-482d-9f58-90f9e0e16546" containerName="main"
Nov 01 15:44:24.652484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.650863    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19a5189-f91f-4561-9fa5-082efe591e41" containerName="main"
Nov 01 15:44:24.653848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.653833    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.709404 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.709363    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1489d8ef-serving-cert\""
Nov 01 15:44:24.742906 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.742875    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkrl\" (UniqueName: \"kubernetes.io/projected/fb17000a-87e8-479c-8dc7-5902f7ea9152-kube-api-access-rzkrl\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.743088 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.742911    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb17000a-87e8-479c-8dc7-5902f7ea9152-openshift-service-ca\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.743088 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.742939    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-metrics-certs\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.743242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.743096    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.761677 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.761654    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"]
Nov 01 15:44:24.843710 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.843679    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkrl\" (UniqueName: \"kubernetes.io/projected/fb17000a-87e8-479c-8dc7-5902f7ea9152-kube-api-access-rzkrl\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.843710 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.843715    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb17000a-87e8-479c-8dc7-5902f7ea9152-openshift-service-ca\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.843924 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.843747    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-metrics-certs\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.843924 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.843843    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.843998 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:44:24.843977    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-serving-cert: secret "maas-api-e2e-ait-1489d8ef-serving-cert" not found
Nov 01 15:44:24.844048 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:44:24.844037    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls podName:fb17000a-87e8-479c-8dc7-5902f7ea9152 nodeName:}" failed. No retries permitted until 2026-11-01 15:44:25.344021651 +0000 UTC m=+5205.509506944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls") pod "maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8" (UID: "fb17000a-87e8-479c-8dc7-5902f7ea9152") : secret "maas-api-e2e-ait-1489d8ef-serving-cert" not found
Nov 01 15:44:24.844353 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.844334    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb17000a-87e8-479c-8dc7-5902f7ea9152-openshift-service-ca\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.846167 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.846147    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-metrics-certs\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:24.901480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:24.901421    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkrl\" (UniqueName: \"kubernetes.io/projected/fb17000a-87e8-479c-8dc7-5902f7ea9152-kube-api-access-rzkrl\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:25.348711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:25.348680    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:25.351018 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:25.350991    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls\") pod \"maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:25.566301 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:25.566270    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:25.928877 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:44:25.928838    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb17000a_87e8_479c_8dc7_5902f7ea9152.slice/crio-11fa34e8e70ad9ebceeb617226ca2d1f6cbd866567fe24ee2272a846854c26f8 WatchSource:0}: Error finding container 11fa34e8e70ad9ebceeb617226ca2d1f6cbd866567fe24ee2272a846854c26f8: Status 404 returned error can't find the container with id 11fa34e8e70ad9ebceeb617226ca2d1f6cbd866567fe24ee2272a846854c26f8
Nov 01 15:44:25.930278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:25.930253    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"]
Nov 01 15:44:25.930417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:25.930373    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:44:26.414359 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:26.414321    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8" event={"ID":"fb17000a-87e8-479c-8dc7-5902f7ea9152","Type":"ContainerStarted","Data":"11fa34e8e70ad9ebceeb617226ca2d1f6cbd866567fe24ee2272a846854c26f8"}
Nov 01 15:44:27.421144 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:27.421112    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8" event={"ID":"fb17000a-87e8-479c-8dc7-5902f7ea9152","Type":"ContainerStarted","Data":"5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b"}
Nov 01 15:44:27.421562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:27.421209    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:27.587196 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:27.587145    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8" podStartSLOduration=2.673405416 podStartE2EDuration="3.587130843s" totalImagesPullingTime="913.725427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:44:27.587116054 +0000 UTC m=+5207.752601346" watchObservedRunningTime="2026-11-01 15:44:27.587130843 +0000 UTC m=+5207.752616138"
Nov 01 15:44:32.173940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:32.173899    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:44:33.433318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:44:33.433292    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:44:35.151767 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:44:35.151727    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:45:05.147839 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:45:05.147795    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:45:09.237821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.237787    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"]
Nov 01 15:45:09.238407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.238061    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8" podUID="fb17000a-87e8-479c-8dc7-5902f7ea9152" containerName="maas-api" containerID="cri-o://5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b" gracePeriod=30
Nov 01 15:45:09.488462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.488405    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:45:09.540666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.540642    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls\") pod \"fb17000a-87e8-479c-8dc7-5902f7ea9152\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") "
Nov 01 15:45:09.540801 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.540685    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-metrics-certs\") pod \"fb17000a-87e8-479c-8dc7-5902f7ea9152\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") "
Nov 01 15:45:09.540801 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.540722    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb17000a-87e8-479c-8dc7-5902f7ea9152-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb17000a-87e8-479c-8dc7-5902f7ea9152-openshift-service-ca\") pod \"fb17000a-87e8-479c-8dc7-5902f7ea9152\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") "
Nov 01 15:45:09.540951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.540929    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb17000a-87e8-479c-8dc7-5902f7ea9152-kube-api-access-rzkrl\" (UniqueName: \"kubernetes.io/projected/fb17000a-87e8-479c-8dc7-5902f7ea9152-kube-api-access-rzkrl\") pod \"fb17000a-87e8-479c-8dc7-5902f7ea9152\" (UID: \"fb17000a-87e8-479c-8dc7-5902f7ea9152\") "
Nov 01 15:45:09.541090 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.541068    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb17000a-87e8-479c-8dc7-5902f7ea9152-openshift-service-ca" pod "fb17000a-87e8-479c-8dc7-5902f7ea9152" (UID: "fb17000a-87e8-479c-8dc7-5902f7ea9152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:45:09.541291 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.541266    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb17000a-87e8-479c-8dc7-5902f7ea9152-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:09.542890 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.542861    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-metrics-certs" pod "fb17000a-87e8-479c-8dc7-5902f7ea9152" (UID: "fb17000a-87e8-479c-8dc7-5902f7ea9152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:45:09.543215 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.543194    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb17000a-87e8-479c-8dc7-5902f7ea9152-kube-api-access-rzkrl" pod "fb17000a-87e8-479c-8dc7-5902f7ea9152" (UID: "fb17000a-87e8-479c-8dc7-5902f7ea9152"). InnerVolumeSpecName "kube-api-access-rzkrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:45:09.543303 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.543269    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls" pod "fb17000a-87e8-479c-8dc7-5902f7ea9152" (UID: "fb17000a-87e8-479c-8dc7-5902f7ea9152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:45:09.615860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.615830    5423 generic.go:356] "Generic (PLEG): container finished" podID="fb17000a-87e8-479c-8dc7-5902f7ea9152" containerID="5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b" exitCode=0
Nov 01 15:45:09.616024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.615893    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"
Nov 01 15:45:09.616024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.615905    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8" event={"ID":"fb17000a-87e8-479c-8dc7-5902f7ea9152","Type":"ContainerDied","Data":"5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b"}
Nov 01 15:45:09.616024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.615956    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8" event={"ID":"fb17000a-87e8-479c-8dc7-5902f7ea9152","Type":"ContainerDied","Data":"11fa34e8e70ad9ebceeb617226ca2d1f6cbd866567fe24ee2272a846854c26f8"}
Nov 01 15:45:09.616024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.615978    5423 scope.go:122] "RemoveContainer" containerID="5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b"
Nov 01 15:45:09.624758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.624733    5423 scope.go:122] "RemoveContainer" containerID="5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b"
Nov 01 15:45:09.625034 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:45:09.625016    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b\": container with ID starting with 5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b not found: ID does not exist" containerID="5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b"
Nov 01 15:45:09.625108 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.625042    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b"} err="failed to get container status \"5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b\": rpc error: code = NotFound desc = could not find container \"5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b\": container with ID starting with 5f2f4a565a420eb0c774438eda1706dc5c71d3724567901b2207ca815822d15b not found: ID does not exist"
Nov 01 15:45:09.641916 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.641895    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzkrl\" (UniqueName: \"kubernetes.io/projected/fb17000a-87e8-479c-8dc7-5902f7ea9152-kube-api-access-rzkrl\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:09.642011 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.641919    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:09.642011 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.641933    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb17000a-87e8-479c-8dc7-5902f7ea9152-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:09.783265 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.783187    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"]
Nov 01 15:45:09.834445 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.834415    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1489d8ef-684fb79cb-57kl8"]
Nov 01 15:45:09.959825 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:09.959789    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb17000a-87e8-479c-8dc7-5902f7ea9152" path="/var/lib/kubelet/pods/fb17000a-87e8-479c-8dc7-5902f7ea9152/volumes"
Nov 01 15:45:13.569521 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:13.569487    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:45:20.828269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:20.828222    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-x48jt"]
Nov 01 15:45:20.828691 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:20.828445    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt" podUID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerName="manager" containerID="cri-o://ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb" gracePeriod=30
Nov 01 15:45:21.081443 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.081367    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 15:45:21.144028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.143976    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-metrics-certs\") pod \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") "
Nov 01 15:45:21.144192 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.144089    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-kube-api-access-h9mb7\" (UniqueName: \"kubernetes.io/projected/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-kube-api-access-h9mb7\") pod \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") "
Nov 01 15:45:21.144192 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.144167    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-webhook-cert\") pod \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\" (UID: \"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc\") "
Nov 01 15:45:21.146177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.146144    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-kube-api-access-h9mb7" pod "1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" (UID: "1e87bc8c-e77e-42fe-ba31-b75c4a5967cc"). InnerVolumeSpecName "kube-api-access-h9mb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:45:21.146177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.146160    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-metrics-certs" pod "1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" (UID: "1e87bc8c-e77e-42fe-ba31-b75c4a5967cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:45:21.146311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.146185    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-webhook-cert" pod "1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" (UID: "1e87bc8c-e77e-42fe-ba31-b75c4a5967cc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:45:21.245868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.245811    5423 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-webhook-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:21.245868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.245862    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:21.245868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.245879    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9mb7\" (UniqueName: \"kubernetes.io/projected/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc-kube-api-access-h9mb7\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:21.669199 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.669165    5423 generic.go:356] "Generic (PLEG): container finished" podID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerID="ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb" exitCode=0
Nov 01 15:45:21.669414 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.669237    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt"
Nov 01 15:45:21.669414 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.669255    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt" event={"ID":"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc","Type":"ContainerDied","Data":"ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb"}
Nov 01 15:45:21.669414 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.669320    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-x48jt" event={"ID":"1e87bc8c-e77e-42fe-ba31-b75c4a5967cc","Type":"ContainerDied","Data":"f864a32444ed82b9f63c45974eec6e0e4d55b414f957a67fc0cd2c5331aed26f"}
Nov 01 15:45:21.669414 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.669350    5423 scope.go:122] "RemoveContainer" containerID="ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb"
Nov 01 15:45:21.679294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.679274    5423 scope.go:122] "RemoveContainer" containerID="8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe"
Nov 01 15:45:21.686670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.686653    5423 scope.go:122] "RemoveContainer" containerID="ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb"
Nov 01 15:45:21.686900 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:45:21.686880    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb\": container with ID starting with ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb not found: ID does not exist" containerID="ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb"
Nov 01 15:45:21.686995 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.686905    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb"} err="failed to get container status \"ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb\": rpc error: code = NotFound desc = could not find container \"ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb\": container with ID starting with ae0c2d00ce303ac8b79adb9d833958084dac6b50b7f79a6e31ae080c7690e8eb not found: ID does not exist"
Nov 01 15:45:21.686995 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.686921    5423 scope.go:122] "RemoveContainer" containerID="8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe"
Nov 01 15:45:21.687184 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:45:21.687168    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe\": container with ID starting with 8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe not found: ID does not exist" containerID="8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe"
Nov 01 15:45:21.687257 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.687186    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe"} err="failed to get container status \"8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe\": rpc error: code = NotFound desc = could not find container \"8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe\": container with ID starting with 8eddf0a9e9a3d015dea916f023bce4bca4e9fb3988fc6a023979947de92eefbe not found: ID does not exist"
Nov 01 15:45:21.835200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.835167    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-x48jt"]
Nov 01 15:45:21.887340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.887307    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-x48jt"]
Nov 01 15:45:21.964378 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:21.964304    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" path="/var/lib/kubelet/pods/1e87bc8c-e77e-42fe-ba31-b75c4a5967cc/volumes"
Nov 01 15:45:23.287198 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287154    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-fx6pq"]
Nov 01 15:45:23.287618 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287553    5423 cpu_manager.go:414] "containerMap: removing container" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy"
Nov 01 15:45:23.287618 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287564    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy"
Nov 01 15:45:23.287618 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287578    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerName="manager"
Nov 01 15:45:23.287618 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287586    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerName="manager"
Nov 01 15:45:23.287618 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287599    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy"
Nov 01 15:45:23.287618 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287607    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0966ad2c-b364-4404-9fdc-479744097176" containerName="istio-proxy"
Nov 01 15:45:23.287618 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287621    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerName="manager"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287625    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerName="manager"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287641    5423 cpu_manager.go:414] "containerMap: removing container" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287648    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287664    5423 cpu_manager.go:414] "containerMap: removing container" podUID="fb17000a-87e8-479c-8dc7-5902f7ea9152" containerName="maas-api"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287671    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb17000a-87e8-479c-8dc7-5902f7ea9152" containerName="maas-api"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287775    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerName="manager"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287790    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f5dd4e-c04a-47ec-8291-5a6565e93bde" containerName="istio-proxy"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287800    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="520751d7-df0b-4875-8e79-9d9703298a93" containerName="istio-proxy"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287811    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb5f2ae-d498-4cde-95be-e7dd1339c347" containerName="istio-proxy"
Nov 01 15:45:23.287850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.287820    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb17000a-87e8-479c-8dc7-5902f7ea9152" containerName="maas-api"
Nov 01 15:45:23.292496 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.292478    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.346152 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.346121    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 15:45:23.370615 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.370589    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-webhook-cert\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.370714 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.370639    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-metrics-certs\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.370762 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.370723    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvn9p\" (UniqueName: \"kubernetes.io/projected/3f0b8404-0bd9-48ae-899a-001902054363-kube-api-access-rvn9p\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.397375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.397339    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lqgws\""
Nov 01 15:45:23.397375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.397360    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 15:45:23.397375 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.397368    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 15:45:23.398766 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.398748    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-fx6pq"]
Nov 01 15:45:23.471345 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.471316    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-webhook-cert\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.471499 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.471366    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-metrics-certs\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.471499 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.471419    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvn9p\" (UniqueName: \"kubernetes.io/projected/3f0b8404-0bd9-48ae-899a-001902054363-kube-api-access-rvn9p\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.473908 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.473892    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-metrics-certs\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.473978 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.473930    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-webhook-cert\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.535250 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.535230    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvn9p\" (UniqueName: \"kubernetes.io/projected/3f0b8404-0bd9-48ae-899a-001902054363-kube-api-access-rvn9p\") pod \"maas-controller-75bf7b5bf6-fx6pq\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") " pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.604565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.604545    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:23.971739 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:45:23.971704    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0b8404_0bd9_48ae_899a_001902054363.slice/crio-0b54dc4a5bba77af0ff353e31f3313166d11017c9e0b764b45ce6750e772c5b6 WatchSource:0}: Error finding container 0b54dc4a5bba77af0ff353e31f3313166d11017c9e0b764b45ce6750e772c5b6: Status 404 returned error can't find the container with id 0b54dc4a5bba77af0ff353e31f3313166d11017c9e0b764b45ce6750e772c5b6
Nov 01 15:45:23.973800 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:23.973781    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-fx6pq"]
Nov 01 15:45:24.690366 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:24.690330    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq" event={"ID":"3f0b8404-0bd9-48ae-899a-001902054363","Type":"ContainerStarted","Data":"0b54dc4a5bba77af0ff353e31f3313166d11017c9e0b764b45ce6750e772c5b6"}
Nov 01 15:45:25.696881 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:25.696843    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq" event={"ID":"3f0b8404-0bd9-48ae-899a-001902054363","Type":"ContainerStarted","Data":"5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b"}
Nov 01 15:45:25.697269 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:25.696986    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:25.869129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:25.869068    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq" podStartSLOduration=1.893263301 podStartE2EDuration="2.86905251s" totalImagesPullingTime="975.789209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:45:25.81759597 +0000 UTC m=+5265.983081267" watchObservedRunningTime="2026-11-01 15:45:25.86905251 +0000 UTC m=+5266.034537806"
Nov 01 15:45:35.147077 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:45:35.147013    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:45:36.703304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:36.703274    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:56.300711 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.300677    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-fx6pq"]
Nov 01 15:45:56.301193 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.300878    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq" podUID="3f0b8404-0bd9-48ae-899a-001902054363" containerName="manager" containerID="cri-o://5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b" gracePeriod=30
Nov 01 15:45:56.542950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.542926    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:56.582241 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.582216    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-metrics-certs\") pod \"3f0b8404-0bd9-48ae-899a-001902054363\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") "
Nov 01 15:45:56.582364 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.582256    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0b8404-0bd9-48ae-899a-001902054363-kube-api-access-rvn9p\" (UniqueName: \"kubernetes.io/projected/3f0b8404-0bd9-48ae-899a-001902054363-kube-api-access-rvn9p\") pod \"3f0b8404-0bd9-48ae-899a-001902054363\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") "
Nov 01 15:45:56.582475 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.582452    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-webhook-cert\") pod \"3f0b8404-0bd9-48ae-899a-001902054363\" (UID: \"3f0b8404-0bd9-48ae-899a-001902054363\") "
Nov 01 15:45:56.584363 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.584335    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-metrics-certs" pod "3f0b8404-0bd9-48ae-899a-001902054363" (UID: "3f0b8404-0bd9-48ae-899a-001902054363"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:45:56.584479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.584372    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-webhook-cert" pod "3f0b8404-0bd9-48ae-899a-001902054363" (UID: "3f0b8404-0bd9-48ae-899a-001902054363"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:45:56.584479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.584415    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0b8404-0bd9-48ae-899a-001902054363-kube-api-access-rvn9p" pod "3f0b8404-0bd9-48ae-899a-001902054363" (UID: "3f0b8404-0bd9-48ae-899a-001902054363"). InnerVolumeSpecName "kube-api-access-rvn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:45:56.683805 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.683764    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:56.683963 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.683808    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvn9p\" (UniqueName: \"kubernetes.io/projected/3f0b8404-0bd9-48ae-899a-001902054363-kube-api-access-rvn9p\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:56.683963 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.683826    5423 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f0b8404-0bd9-48ae-899a-001902054363-webhook-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:56.838267 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.838186    5423 generic.go:356] "Generic (PLEG): container finished" podID="3f0b8404-0bd9-48ae-899a-001902054363" containerID="5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b" exitCode=0
Nov 01 15:45:56.838267 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.838245    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq"
Nov 01 15:45:56.838267 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.838259    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq" event={"ID":"3f0b8404-0bd9-48ae-899a-001902054363","Type":"ContainerDied","Data":"5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b"}
Nov 01 15:45:56.838560 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.838303    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-fx6pq" event={"ID":"3f0b8404-0bd9-48ae-899a-001902054363","Type":"ContainerDied","Data":"0b54dc4a5bba77af0ff353e31f3313166d11017c9e0b764b45ce6750e772c5b6"}
Nov 01 15:45:56.838560 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.838325    5423 scope.go:122] "RemoveContainer" containerID="5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b"
Nov 01 15:45:56.847825 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.847805    5423 scope.go:122] "RemoveContainer" containerID="5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b"
Nov 01 15:45:56.848098 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:45:56.848078    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b\": container with ID starting with 5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b not found: ID does not exist" containerID="5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b"
Nov 01 15:45:56.848164 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:56.848112    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b"} err="failed to get container status \"5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b\": rpc error: code = NotFound desc = could not find container \"5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b\": container with ID starting with 5b79b55b354be881cd3fbbc9c6d288d159dff4a684ef372d1884d2a24aca049b not found: ID does not exist"
Nov 01 15:45:57.003976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:57.003940    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-fx6pq"]
Nov 01 15:45:57.055218 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:57.055177    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-fx6pq"]
Nov 01 15:45:57.959841 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:57.959805    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0b8404-0bd9-48ae-899a-001902054363" path="/var/lib/kubelet/pods/3f0b8404-0bd9-48ae-899a-001902054363/volumes"
Nov 01 15:45:58.423369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.423338    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-4vbfk"]
Nov 01 15:45:58.423770 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.423755    5423 cpu_manager.go:414] "containerMap: removing container" podUID="3f0b8404-0bd9-48ae-899a-001902054363" containerName="manager"
Nov 01 15:45:58.423848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.423770    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0b8404-0bd9-48ae-899a-001902054363" containerName="manager"
Nov 01 15:45:58.423892 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.423853    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0b8404-0bd9-48ae-899a-001902054363" containerName="manager"
Nov 01 15:45:58.423892 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.423864    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e87bc8c-e77e-42fe-ba31-b75c4a5967cc" containerName="manager"
Nov 01 15:45:58.428461 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.428443    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.501450 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.501415    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0d579d-61a9-4213-a72d-dcf2a340496b-webhook-cert\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.501611 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.501472    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl587\" (UniqueName: \"kubernetes.io/projected/1c0d579d-61a9-4213-a72d-dcf2a340496b-kube-api-access-kl587\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.501611 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.501526    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0d579d-61a9-4213-a72d-dcf2a340496b-metrics-certs\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.533318 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.533278    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lqgws\""
Nov 01 15:45:58.533513 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.533300    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 15:45:58.533513 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.533302    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 15:45:58.533513 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.533326    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 15:45:58.534658 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.534633    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-4vbfk"]
Nov 01 15:45:58.602809 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.602774    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0d579d-61a9-4213-a72d-dcf2a340496b-webhook-cert\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.602976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.602838    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl587\" (UniqueName: \"kubernetes.io/projected/1c0d579d-61a9-4213-a72d-dcf2a340496b-kube-api-access-kl587\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.602976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.602863    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0d579d-61a9-4213-a72d-dcf2a340496b-metrics-certs\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.605570 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.605545    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0d579d-61a9-4213-a72d-dcf2a340496b-webhook-cert\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.605701 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.605545    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0d579d-61a9-4213-a72d-dcf2a340496b-metrics-certs\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.660647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.660610    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl587\" (UniqueName: \"kubernetes.io/projected/1c0d579d-61a9-4213-a72d-dcf2a340496b-kube-api-access-kl587\") pod \"maas-controller-75bf7b5bf6-4vbfk\" (UID: \"1c0d579d-61a9-4213-a72d-dcf2a340496b\") " pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:58.741869 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:58.741777    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:45:59.105031 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:45:59.104994    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0d579d_61a9_4213_a72d_dcf2a340496b.slice/crio-d2061474fea77505240904961bf4e7f013b4f69854deb825e7fe2f9be1bee5c3 WatchSource:0}: Error finding container d2061474fea77505240904961bf4e7f013b4f69854deb825e7fe2f9be1bee5c3: Status 404 returned error can't find the container with id d2061474fea77505240904961bf4e7f013b4f69854deb825e7fe2f9be1bee5c3
Nov 01 15:45:59.106782 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:59.106749    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75bf7b5bf6-4vbfk"]
Nov 01 15:45:59.858982 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:45:59.858937    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk" event={"ID":"1c0d579d-61a9-4213-a72d-dcf2a340496b","Type":"ContainerStarted","Data":"d2061474fea77505240904961bf4e7f013b4f69854deb825e7fe2f9be1bee5c3"}
Nov 01 15:46:00.108097 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.108066    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"]
Nov 01 15:46:00.111461 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.111409    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.216535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.216491    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"]
Nov 01 15:46:00.216987 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.216962    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.217127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.216997    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.217127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.217023    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.217127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.217048    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.217278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.217212    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.217321 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.217268    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqkb\" (UniqueName: \"kubernetes.io/projected/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-kube-api-access-bwqkb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.217366 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.217331    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.217457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.217367    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.217457 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.217445    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318213 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318175    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318224    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318269    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318598 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318480    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318598 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318512    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318598 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318538    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318598 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318564    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318598    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318629    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318652    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.318806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318655    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqkb\" (UniqueName: \"kubernetes.io/projected/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-kube-api-access-bwqkb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.319004 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318878    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.319004 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.318943    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.319241 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.319223    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.320857 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.320834    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.320987 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.320970    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.377599 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.377533    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqkb\" (UniqueName: \"kubernetes.io/projected/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-kube-api-access-bwqkb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.377599 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.377548    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gkt8b\" (UID: \"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.424735 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.424701    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:00.837946 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:46:00.837911    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bd3a7d_cb50_4c9e_b20c_c137bdbce8c7.slice/crio-70507edc043a5707b7cdb45af6bb2836b23728d0172264b6fc3ca63444d62b69 WatchSource:0}: Error finding container 70507edc043a5707b7cdb45af6bb2836b23728d0172264b6fc3ca63444d62b69: Status 404 returned error can't find the container with id 70507edc043a5707b7cdb45af6bb2836b23728d0172264b6fc3ca63444d62b69
Nov 01 15:46:00.839320 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.839292    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"]
Nov 01 15:46:00.840516 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.840477    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:46:00.840582 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.840568    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:46:00.840630 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.840617    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:46:00.864777 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.864739    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk" event={"ID":"1c0d579d-61a9-4213-a72d-dcf2a340496b","Type":"ContainerStarted","Data":"0ee2a4c0f219446015024af6c5369b591e5ed78a62122657544a0228ea6db96e"}
Nov 01 15:46:00.864889 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.864799    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:46:00.865964 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.865940    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" event={"ID":"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7","Type":"ContainerStarted","Data":"70507edc043a5707b7cdb45af6bb2836b23728d0172264b6fc3ca63444d62b69"}
Nov 01 15:46:00.981774 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:00.981710    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk" podStartSLOduration=2.132879852 podStartE2EDuration="2.981691614s" totalImagesPullingTime="848.811762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:46:00.980248856 +0000 UTC m=+5301.145734156" watchObservedRunningTime="2026-11-01 15:46:00.981691614 +0000 UTC m=+5301.147176913"
Nov 01 15:46:01.871812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:01.871763    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" event={"ID":"35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7","Type":"ContainerStarted","Data":"579fb461b82be3910e4cb3774531bdb7d2327632df0bfc59d01ddc6a0f249f48"}
Nov 01 15:46:02.091905 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:02.091854    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podStartSLOduration=2.091840125 podStartE2EDuration="2.091840125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:46:02.040746294 +0000 UTC m=+5302.206231590" watchObservedRunningTime="2026-11-01 15:46:02.091840125 +0000 UTC m=+5302.257325422"
Nov 01 15:46:02.425738 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:02.425701    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:02.427177 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:02.427149    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gkt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 01 15:46:02.427314 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:02.427199    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podUID="35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 01 15:46:03.425853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:03.425818    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gkt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 01 15:46:03.426253 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:03.425871    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podUID="35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 01 15:46:04.425926 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:04.425886    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gkt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 01 15:46:04.426320 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:04.425946    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podUID="35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 01 15:46:05.148635 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:46:05.148600    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:46:05.425927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:05.425846    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gkt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 01 15:46:05.425927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:05.425903    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podUID="35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 01 15:46:06.425827 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:06.425796    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gkt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 01 15:46:06.426004 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:06.425850    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podUID="35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 01 15:46:07.425110 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:07.425074    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gkt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 01 15:46:07.425631 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:07.425133    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podUID="35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 01 15:46:08.879913 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:08.879877    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:08.880434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:08.880275    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:08.880966 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:08.880940    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b"
Nov 01 15:46:11.874224 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:11.874196    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75bf7b5bf6-4vbfk"
Nov 01 15:46:25.069689 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:25.069608    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:46:35.148312 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:46:35.148275    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:46:43.940501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:43.940464    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:46:45.131422 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.131369    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"]
Nov 01 15:46:45.136886 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.136861    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.229294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229260    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.229485 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229379    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.229485 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229454    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf4sl\" (UniqueName: \"kubernetes.io/projected/c3e03db8-a460-41e6-8eb4-a044121bd5dc-kube-api-access-bf4sl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.229625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229543    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.229625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229585    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.229853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229658    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.229853 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229705    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.230032 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229847    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.230032 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.229884    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.331294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.331242    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.331583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.331556    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.331757 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.331737    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.331920 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.331904    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.332688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332023    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.332688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332040    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.332688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332067    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.332688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332106    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.332688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332150    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.332688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332230    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.332688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332275    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf4sl\" (UniqueName: \"kubernetes.io/projected/c3e03db8-a460-41e6-8eb4-a044121bd5dc-kube-api-access-bf4sl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.332688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332345    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.333234 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.332807    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.333234 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.333039    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.334898 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.334852    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.335018 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.334936    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.372176 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.372149    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"]
Nov 01 15:46:45.390334 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.390276    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e03db8-a460-41e6-8eb4-a044121bd5dc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.390443 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.390415    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf4sl\" (UniqueName: \"kubernetes.io/projected/c3e03db8-a460-41e6-8eb4-a044121bd5dc-kube-api-access-bf4sl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pw4lq\" (UID: \"c3e03db8-a460-41e6-8eb4-a044121bd5dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.454249 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.454214    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:45.870850 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:46:45.870814    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e03db8_a460_41e6_8eb4_a044121bd5dc.slice/crio-382ce888a8cc754a75c5cb058182ad697a4773a98f3664119457f90349012ae5 WatchSource:0}: Error finding container 382ce888a8cc754a75c5cb058182ad697a4773a98f3664119457f90349012ae5: Status 404 returned error can't find the container with id 382ce888a8cc754a75c5cb058182ad697a4773a98f3664119457f90349012ae5
Nov 01 15:46:45.872687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.872663    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"]
Nov 01 15:46:45.873136 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.873103    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:46:45.873245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.873174    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:46:45.873245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:45.873215    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:46:46.065492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:46.065418    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" event={"ID":"c3e03db8-a460-41e6-8eb4-a044121bd5dc","Type":"ContainerStarted","Data":"d938477c3cee93b98650cc9d03456a2aa5243e8946f1eb9927b0eb480c3a0159"}
Nov 01 15:46:46.065492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:46.065489    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" event={"ID":"c3e03db8-a460-41e6-8eb4-a044121bd5dc","Type":"ContainerStarted","Data":"382ce888a8cc754a75c5cb058182ad697a4773a98f3664119457f90349012ae5"}
Nov 01 15:46:46.454522 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:46.454488    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:46.456066 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:46.456032    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 01 15:46:46.456211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:46.456084    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 01 15:46:47.454698 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:47.454666    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 01 15:46:47.455121 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:47.454712    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 01 15:46:48.454611 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:48.454579    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 01 15:46:48.454806 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:48.454629    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 01 15:46:49.454747 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:49.454700    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 01 15:46:49.454972 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:49.454756    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 01 15:46:50.455365 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:50.455323    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 01 15:46:50.456022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:50.455989    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 01 15:46:51.454605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:51.454564    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 01 15:46:51.454820 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:51.454623    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 01 15:46:52.454780 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:52.454746    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 01 15:46:52.455297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:52.454802    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 01 15:46:53.064737 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:53.064675    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podStartSLOduration=8.064661495 podStartE2EDuration="8.064661495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:46:46.185260533 +0000 UTC m=+5346.350745834" watchObservedRunningTime="2026-11-01 15:46:53.064661495 +0000 UTC m=+5353.230146791"
Nov 01 15:46:53.065574 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:53.065558    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:46:54.455471 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:54.455426    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:46:54.455895 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:54.455493    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:46:55.496581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:55.496537    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:46:55.497044 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:55.496615    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:46:56.009950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:56.009913    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:56.010335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:56.010307    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:46:56.011098 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:46:56.011076    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq"
Nov 01 15:47:01.960746 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:01.960718    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:47:05.148071 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:47:05.148035    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:47:09.946352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:09.946318    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:47:35.148549 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:47:35.148512    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:47:43.215158 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:43.215128    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:47:43.234999 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:43.234970    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:47:46.969998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:46.969960    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:47:49.507901 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.507809    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"]
Nov 01 15:47:49.511540 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.511518    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.576424 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.576366    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-tmp-dir\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.576606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.576482    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-dshm\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.576606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.576530    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-home\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.576606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.576562    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d526cf9-c346-46d2-89c9-58b180f04309-tls-certs\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.576732 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.576696    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvmv\" (UniqueName: \"kubernetes.io/projected/9d526cf9-c346-46d2-89c9-58b180f04309-kube-api-access-5nvmv\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.576775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.576740    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-model-cache\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.617662 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.617639    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4dbaa3d3-kserve-self-signed-certs\""
Nov 01 15:47:49.619241 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.619216    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"]
Nov 01 15:47:49.678276 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678246    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-dshm\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.678490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678287    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-home\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.678490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678310    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d526cf9-c346-46d2-89c9-58b180f04309-tls-certs\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.678490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678356    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvmv\" (UniqueName: \"kubernetes.io/projected/9d526cf9-c346-46d2-89c9-58b180f04309-kube-api-access-5nvmv\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.678490 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678422    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-model-cache\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.678739 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678504    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-tmp-dir\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.678739 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678661    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-home\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.678848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678807    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-tmp-dir\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.678894 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.678844    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-model-cache\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.681197 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.681175    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-dshm\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.681366 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.681350    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d526cf9-c346-46d2-89c9-58b180f04309-tls-certs\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.736319 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.736292    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvmv\" (UniqueName: \"kubernetes.io/projected/9d526cf9-c346-46d2-89c9-58b180f04309-kube-api-access-5nvmv\") pod \"e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:49.825874 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:49.825847    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:50.193073 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:47:50.193036    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d526cf9_c346_46d2_89c9_58b180f04309.slice/crio-7c7dfd032e0ba7f8e177d49e3d79c79b5dd712939419641fd5797eca23f5b6f7 WatchSource:0}: Error finding container 7c7dfd032e0ba7f8e177d49e3d79c79b5dd712939419641fd5797eca23f5b6f7: Status 404 returned error can't find the container with id 7c7dfd032e0ba7f8e177d49e3d79c79b5dd712939419641fd5797eca23f5b6f7
Nov 01 15:47:50.194573 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:50.194546    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"]
Nov 01 15:47:50.410037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:50.409961    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6" event={"ID":"9d526cf9-c346-46d2-89c9-58b180f04309","Type":"ContainerStarted","Data":"6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde"}
Nov 01 15:47:50.410037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:50.410014    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6" event={"ID":"9d526cf9-c346-46d2-89c9-58b180f04309","Type":"ContainerStarted","Data":"7c7dfd032e0ba7f8e177d49e3d79c79b5dd712939419641fd5797eca23f5b6f7"}
Nov 01 15:47:50.525765 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:50.525677    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6" podStartSLOduration=1.525664677 podStartE2EDuration="1.525664677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:50.525485826 +0000 UTC m=+5410.690971122" watchObservedRunningTime="2026-11-01 15:47:50.525664677 +0000 UTC m=+5410.691149966"
Nov 01 15:47:50.826505 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:50.826475    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:59.826123 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:59.826086    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:59.838587 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:59.838565    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:47:59.860163 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:47:59.860136    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:48:00.481619 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:48:00.481592    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:48:05.148133 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:48:05.148094    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:48:32.580170 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:48:32.580134    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:48:35.188807 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:48:35.188764    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:48:58.709262 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:48:58.709226    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:48:58.710869 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:48:58.710844    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"]
Nov 01 15:49:05.190673 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:49:05.190638    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:49:09.182341 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:09.182307    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:49:13.731327 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:13.731284    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6" podUID="9d526cf9-c346-46d2-89c9-58b180f04309" containerName="main" containerID="cri-o://6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde" gracePeriod=45
Nov 01 15:49:13.993477 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:13.993420    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:49:14.156856 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.156821    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-dshm\") pod \"9d526cf9-c346-46d2-89c9-58b180f04309\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") "
Nov 01 15:49:14.157022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.156878    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d526cf9-c346-46d2-89c9-58b180f04309-kube-api-access-5nvmv\" (UniqueName: \"kubernetes.io/projected/9d526cf9-c346-46d2-89c9-58b180f04309-kube-api-access-5nvmv\") pod \"9d526cf9-c346-46d2-89c9-58b180f04309\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") "
Nov 01 15:49:14.157022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.156918    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-tmp-dir\") pod \"9d526cf9-c346-46d2-89c9-58b180f04309\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") "
Nov 01 15:49:14.157022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.156980    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-model-cache\") pod \"9d526cf9-c346-46d2-89c9-58b180f04309\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") "
Nov 01 15:49:14.157022 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.157011    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d526cf9-c346-46d2-89c9-58b180f04309-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d526cf9-c346-46d2-89c9-58b180f04309-tls-certs\") pod \"9d526cf9-c346-46d2-89c9-58b180f04309\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") "
Nov 01 15:49:14.157220 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.157041    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-home\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-home\") pod \"9d526cf9-c346-46d2-89c9-58b180f04309\" (UID: \"9d526cf9-c346-46d2-89c9-58b180f04309\") "
Nov 01 15:49:14.157295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.157256    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-tmp-dir" pod "9d526cf9-c346-46d2-89c9-58b180f04309" (UID: "9d526cf9-c346-46d2-89c9-58b180f04309"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:49:14.157295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.157275    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-model-cache" pod "9d526cf9-c346-46d2-89c9-58b180f04309" (UID: "9d526cf9-c346-46d2-89c9-58b180f04309"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:49:14.157433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.157414    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-home" pod "9d526cf9-c346-46d2-89c9-58b180f04309" (UID: "9d526cf9-c346-46d2-89c9-58b180f04309"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:49:14.157493 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.157432    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:14.157493 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.157451    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:14.159174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.159154    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-dshm" pod "9d526cf9-c346-46d2-89c9-58b180f04309" (UID: "9d526cf9-c346-46d2-89c9-58b180f04309"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:49:14.159283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.159184    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d526cf9-c346-46d2-89c9-58b180f04309-tls-certs" pod "9d526cf9-c346-46d2-89c9-58b180f04309" (UID: "9d526cf9-c346-46d2-89c9-58b180f04309"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:49:14.159283 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.159193    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d526cf9-c346-46d2-89c9-58b180f04309-kube-api-access-5nvmv" pod "9d526cf9-c346-46d2-89c9-58b180f04309" (UID: "9d526cf9-c346-46d2-89c9-58b180f04309"). InnerVolumeSpecName "kube-api-access-5nvmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:49:14.258340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.258257    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d526cf9-c346-46d2-89c9-58b180f04309-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:14.258340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.258285    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:14.258340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.258297    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d526cf9-c346-46d2-89c9-58b180f04309-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:14.258340 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.258311    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvmv\" (UniqueName: \"kubernetes.io/projected/9d526cf9-c346-46d2-89c9-58b180f04309-kube-api-access-5nvmv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:14.831078 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.831024    5423 generic.go:356] "Generic (PLEG): container finished" podID="9d526cf9-c346-46d2-89c9-58b180f04309" containerID="6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde" exitCode=0
Nov 01 15:49:14.831595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.831090    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"
Nov 01 15:49:14.831595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.831107    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6" event={"ID":"9d526cf9-c346-46d2-89c9-58b180f04309","Type":"ContainerDied","Data":"6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde"}
Nov 01 15:49:14.831595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.831153    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6" event={"ID":"9d526cf9-c346-46d2-89c9-58b180f04309","Type":"ContainerDied","Data":"7c7dfd032e0ba7f8e177d49e3d79c79b5dd712939419641fd5797eca23f5b6f7"}
Nov 01 15:49:14.831595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.831170    5423 scope.go:122] "RemoveContainer" containerID="6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde"
Nov 01 15:49:14.839917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.839898    5423 scope.go:122] "RemoveContainer" containerID="6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde"
Nov 01 15:49:14.840194 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:49:14.840173    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde\": container with ID starting with 6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde not found: ID does not exist" containerID="6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde"
Nov 01 15:49:14.840236 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.840204    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde"} err="failed to get container status \"6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde\": rpc error: code = NotFound desc = could not find container \"6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde\": container with ID starting with 6ebc8de1ed78780bdb9eb70ff4f205b8eef7b780f7b47599f64e319b5ea1dbde not found: ID does not exist"
Nov 01 15:49:14.996592 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:14.996566    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"]
Nov 01 15:49:15.047694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:15.047666    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-5966fbc59f-q6gt6"]
Nov 01 15:49:15.958649 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:15.958615    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d526cf9-c346-46d2-89c9-58b180f04309" path="/var/lib/kubelet/pods/9d526cf9-c346-46d2-89c9-58b180f04309/volumes"
Nov 01 15:49:17.071778 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.071743    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"]
Nov 01 15:49:17.072282 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.072252    5423 cpu_manager.go:414] "containerMap: removing container" podUID="9d526cf9-c346-46d2-89c9-58b180f04309" containerName="main"
Nov 01 15:49:17.072354 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.072285    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d526cf9-c346-46d2-89c9-58b180f04309" containerName="main"
Nov 01 15:49:17.072442 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.072430    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d526cf9-c346-46d2-89c9-58b180f04309" containerName="main"
Nov 01 15:49:17.076860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.076844    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.131514 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.131480    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-9c2ba6-openshift-default-dockercfg-wm7zs\""
Nov 01 15:49:17.184235 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.184204    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"]
Nov 01 15:49:17.185275 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.184409    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r588\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-kube-api-access-7r588\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.185435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.185310    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845f9010-0051-4025-88b2-8269895a692d-istiod-ca-cert\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.185435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.185379    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845f9010-0051-4025-88b2-8269895a692d-istio-podinfo\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.185568 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.185432    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-data\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.185568 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.185473    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-socket\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.185568 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.185508    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-istio-token\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.185568 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.185563    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-envoy\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.185777 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.185606    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-credential-socket\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.185777 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.185659    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-certs\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286288 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286250    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r588\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-kube-api-access-7r588\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286305    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845f9010-0051-4025-88b2-8269895a692d-istiod-ca-cert\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286348    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845f9010-0051-4025-88b2-8269895a692d-istio-podinfo\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286379    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-data\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286426    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-socket\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286463    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-istio-token\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286531 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286513    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-envoy\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286878 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286550    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-credential-socket\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286878 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286597    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-certs\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.286992 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286875    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-data\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.287075 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.286986    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-socket\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.287075 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.287017    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-certs\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.287160 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.287076    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-credential-socket\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.287160 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.287144    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845f9010-0051-4025-88b2-8269895a692d-istiod-ca-cert\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.288880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.288855    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-envoy\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.289046 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.289029    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845f9010-0051-4025-88b2-8269895a692d-istio-podinfo\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.345524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.345457    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-istio-token\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.345648 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.345550    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r588\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-kube-api-access-7r588\") pod \"e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") " pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.401999 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.401961    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:17.847416 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.845309    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"]
Nov 01 15:49:17.862555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.861461    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:49:17.862555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.861546    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:49:17.862555 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:17.861593    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:49:17.868655 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:49:17.864523    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845f9010_0051_4025_88b2_8269895a692d.slice/crio-06acca8bdcc37ca6654b229e4e44c95d05e21c40ecf45b232961023460d1cd7c WatchSource:0}: Error finding container 06acca8bdcc37ca6654b229e4e44c95d05e21c40ecf45b232961023460d1cd7c: Status 404 returned error can't find the container with id 06acca8bdcc37ca6654b229e4e44c95d05e21c40ecf45b232961023460d1cd7c
Nov 01 15:49:18.880716 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:18.880681    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" event={"ID":"845f9010-0051-4025-88b2-8269895a692d","Type":"ContainerStarted","Data":"7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989"}
Nov 01 15:49:18.881137 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:18.880733    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" event={"ID":"845f9010-0051-4025-88b2-8269895a692d","Type":"ContainerStarted","Data":"06acca8bdcc37ca6654b229e4e44c95d05e21c40ecf45b232961023460d1cd7c"}
Nov 01 15:49:19.112934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:19.112857    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" podStartSLOduration=2.112835996 podStartE2EDuration="2.112835996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:19.05195755 +0000 UTC m=+5499.217442849" watchObservedRunningTime="2026-11-01 15:49:19.112835996 +0000 UTC m=+5499.278321285"
Nov 01 15:49:19.402349 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:19.402261    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:19.406998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:19.406978    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:19.885610 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:19.885582    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:19.886574 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:19.886549    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:25.305178 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.305141    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"]
Nov 01 15:49:25.309057 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.309035    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:25.336628 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.336602    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644dc96f-9456-4358-b65e-30a3ee01e354-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5\" (UID: \"644dc96f-9456-4358-b65e-30a3ee01e354\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:25.336739 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.336655    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5xfb\" (UniqueName: \"kubernetes.io/projected/644dc96f-9456-4358-b65e-30a3ee01e354-kube-api-access-z5xfb\") pod \"payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5\" (UID: \"644dc96f-9456-4358-b65e-30a3ee01e354\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:25.421692 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.421660    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9c2ba6\""
Nov 01 15:49:25.421944 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.421659    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bebfaf8e-dockercfg-4k5mh\""
Nov 01 15:49:25.424342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.424313    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"]
Nov 01 15:49:25.437453 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.437424    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5xfb\" (UniqueName: \"kubernetes.io/projected/644dc96f-9456-4358-b65e-30a3ee01e354-kube-api-access-z5xfb\") pod \"payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5\" (UID: \"644dc96f-9456-4358-b65e-30a3ee01e354\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:25.438099 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.438073    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644dc96f-9456-4358-b65e-30a3ee01e354-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5\" (UID: \"644dc96f-9456-4358-b65e-30a3ee01e354\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:25.438989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.438965    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644dc96f-9456-4358-b65e-30a3ee01e354-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5\" (UID: \"644dc96f-9456-4358-b65e-30a3ee01e354\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:25.467602 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.467577    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"]
Nov 01 15:49:25.471149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.471133    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:25.497847 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.497830    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5xfb\" (UniqueName: \"kubernetes.io/projected/644dc96f-9456-4358-b65e-30a3ee01e354-kube-api-access-z5xfb\") pod \"payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5\" (UID: \"644dc96f-9456-4358-b65e-30a3ee01e354\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:25.527684 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.527660    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"]
Nov 01 15:49:25.538679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.538655    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grnmr\" (UniqueName: \"kubernetes.io/projected/4982e142-162e-41bd-8029-ce8196ec44cc-kube-api-access-grnmr\") pod \"payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97\" (UID: \"4982e142-162e-41bd-8029-ce8196ec44cc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:25.538824 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.538711    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982e142-162e-41bd-8029-ce8196ec44cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97\" (UID: \"4982e142-162e-41bd-8029-ce8196ec44cc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:25.622449 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.622422    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:25.639702 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.639672    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982e142-162e-41bd-8029-ce8196ec44cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97\" (UID: \"4982e142-162e-41bd-8029-ce8196ec44cc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:25.639864 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.639775    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grnmr\" (UniqueName: \"kubernetes.io/projected/4982e142-162e-41bd-8029-ce8196ec44cc-kube-api-access-grnmr\") pod \"payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97\" (UID: \"4982e142-162e-41bd-8029-ce8196ec44cc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:25.640364 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.640343    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982e142-162e-41bd-8029-ce8196ec44cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97\" (UID: \"4982e142-162e-41bd-8029-ce8196ec44cc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:25.697606 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.697543    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grnmr\" (UniqueName: \"kubernetes.io/projected/4982e142-162e-41bd-8029-ce8196ec44cc-kube-api-access-grnmr\") pod \"payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97\" (UID: \"4982e142-162e-41bd-8029-ce8196ec44cc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:25.783960 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.783930    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:25.993438 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:25.993415    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"]
Nov 01 15:49:25.995359 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:49:25.995331    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644dc96f_9456_4358_b65e_30a3ee01e354.slice/crio-d996f8023833a83e16ef865599dcd6d8b4db24f2a9b71782b2d3cbc89cdff475 WatchSource:0}: Error finding container d996f8023833a83e16ef865599dcd6d8b4db24f2a9b71782b2d3cbc89cdff475: Status 404 returned error can't find the container with id d996f8023833a83e16ef865599dcd6d8b4db24f2a9b71782b2d3cbc89cdff475
Nov 01 15:49:26.155119 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:26.155025    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"]
Nov 01 15:49:26.157957 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:49:26.157928    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4982e142_162e_41bd_8029_ce8196ec44cc.slice/crio-f7e52c32b07f877e6a8010484d754ffcd3855f1f143a0a5adc9baa5fe75048a6 WatchSource:0}: Error finding container f7e52c32b07f877e6a8010484d754ffcd3855f1f143a0a5adc9baa5fe75048a6: Status 404 returned error can't find the container with id f7e52c32b07f877e6a8010484d754ffcd3855f1f143a0a5adc9baa5fe75048a6
Nov 01 15:49:26.920846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:26.920808    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97" event={"ID":"4982e142-162e-41bd-8029-ce8196ec44cc","Type":"ContainerStarted","Data":"b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934"}
Nov 01 15:49:26.921342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:26.920871    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97" event={"ID":"4982e142-162e-41bd-8029-ce8196ec44cc","Type":"ContainerStarted","Data":"f7e52c32b07f877e6a8010484d754ffcd3855f1f143a0a5adc9baa5fe75048a6"}
Nov 01 15:49:26.921342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:26.920900    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:26.922347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:26.922325    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5" event={"ID":"644dc96f-9456-4358-b65e-30a3ee01e354","Type":"ContainerStarted","Data":"1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5"}
Nov 01 15:49:26.922489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:26.922370    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5" event={"ID":"644dc96f-9456-4358-b65e-30a3ee01e354","Type":"ContainerStarted","Data":"d996f8023833a83e16ef865599dcd6d8b4db24f2a9b71782b2d3cbc89cdff475"}
Nov 01 15:49:26.922489 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:26.922471    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:27.036824 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.036770    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97" podStartSLOduration=2.036757652 podStartE2EDuration="2.036757652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:27.035873243 +0000 UTC m=+5507.201358540" watchObservedRunningTime="2026-11-01 15:49:27.036757652 +0000 UTC m=+5507.202242949"
Nov 01 15:49:27.157439 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.156061    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5" podStartSLOduration=2.156039085 podStartE2EDuration="2.156039085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:27.15307543 +0000 UTC m=+5507.318560729" watchObservedRunningTime="2026-11-01 15:49:27.156039085 +0000 UTC m=+5507.321524383"
Nov 01 15:49:27.260737 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.260656    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"]
Nov 01 15:49:27.275155 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.275088    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.321183 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.321151    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"]
Nov 01 15:49:27.331203 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.331174    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-9c2ba6-serving-cert\""
Nov 01 15:49:27.355965 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.355932    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.356086 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.355998    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f84d1a-1a42-4e4a-aab5-a015f00420d3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.356132 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.356103    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.356176 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.356152    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/85f84d1a-1a42-4e4a-aab5-a015f00420d3-kube-api-access-p9chl\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.456673 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.456642    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.456818 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.456703    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f84d1a-1a42-4e4a-aab5-a015f00420d3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.456818 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.456738    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.456818 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.456768    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/85f84d1a-1a42-4e4a-aab5-a015f00420d3-kube-api-access-p9chl\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.456968 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:49:27.456841    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-serving-cert: secret "maas-api-e2e-worker-single-main-9c2ba6-serving-cert" not found
Nov 01 15:49:27.456968 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:49:27.456895    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls podName:85f84d1a-1a42-4e4a-aab5-a015f00420d3 nodeName:}" failed. No retries permitted until 2026-11-01 15:49:27.956880927 +0000 UTC m=+5508.122366203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls") pod "maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft" (UID: "85f84d1a-1a42-4e4a-aab5-a015f00420d3") : secret "maas-api-e2e-worker-single-main-9c2ba6-serving-cert" not found
Nov 01 15:49:27.457412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.457366    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f84d1a-1a42-4e4a-aab5-a015f00420d3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.459116 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.459099    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.515478 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.515408    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/85f84d1a-1a42-4e4a-aab5-a015f00420d3-kube-api-access-p9chl\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.962751 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.962719    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:27.965165 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:27.965141    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:28.194506 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:28.194473    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:28.566495 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:28.566465    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"]
Nov 01 15:49:28.569095 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:49:28.569057    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f84d1a_1a42_4e4a_aab5_a015f00420d3.slice/crio-4d2ceb5f6635e44e5d7fac9014457dfbf1af8a3764481a2d26028f52967b39fe WatchSource:0}: Error finding container 4d2ceb5f6635e44e5d7fac9014457dfbf1af8a3764481a2d26028f52967b39fe: Status 404 returned error can't find the container with id 4d2ceb5f6635e44e5d7fac9014457dfbf1af8a3764481a2d26028f52967b39fe
Nov 01 15:49:28.570475 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:28.570454    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:49:28.933826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:28.933747    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft" event={"ID":"85f84d1a-1a42-4e4a-aab5-a015f00420d3","Type":"ContainerStarted","Data":"4d2ceb5f6635e44e5d7fac9014457dfbf1af8a3764481a2d26028f52967b39fe"}
Nov 01 15:49:29.939938 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:29.939904    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft" event={"ID":"85f84d1a-1a42-4e4a-aab5-a015f00420d3","Type":"ContainerStarted","Data":"dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278"}
Nov 01 15:49:29.940423 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:29.940054    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:30.057186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:30.057128    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft" podStartSLOduration=2.166151613 podStartE2EDuration="3.057113097s" totalImagesPullingTime="890.961484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:30.056418547 +0000 UTC m=+5510.221903841" watchObservedRunningTime="2026-11-01 15:49:30.057113097 +0000 UTC m=+5510.222598393"
Nov 01 15:49:34.777312 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:34.777272    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:49:35.149512 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:49:35.149472    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:49:35.950758 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:35.950732    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:49:37.930098 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:37.930068    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:49:37.930507 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:37.930131    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:49:45.021819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:45.021784    5423 generic.go:356] "Generic (PLEG): container finished" podID="845f9010-0051-4025-88b2-8269895a692d" containerID="7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989" exitCode=0
Nov 01 15:49:45.022242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:45.021857    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" event={"ID":"845f9010-0051-4025-88b2-8269895a692d","Type":"ContainerDied","Data":"7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989"}
Nov 01 15:49:45.022242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:45.022180    5423 scope.go:122] "RemoveContainer" containerID="7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989"
Nov 01 15:49:45.023307 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:45.023278    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:49:45.023371 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:45.023345    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:49:45.023432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:45.023382    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:49:46.028926 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:46.028894    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" event={"ID":"845f9010-0051-4025-88b2-8269895a692d","Type":"ContainerStarted","Data":"d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d"}
Nov 01 15:49:46.419412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:46.416631    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:46.419635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:46.419495    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 15:49:46.419635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:46.419550    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 15:49:47.415409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:47.407787    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 15:49:47.415409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:47.407853    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 15:49:48.416410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:48.415757    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 15:49:48.416410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:48.415832    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 15:49:49.416409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:49.412827    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 15:49:49.416409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:49.412893    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 15:49:49.886370 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:49.886338    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:50.407805 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:50.407764    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 15:49:50.408192 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:50.408160    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 15:49:51.406849 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:51.406820    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:51.407871 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:51.407850    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:49:54.778553 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:49:54.778519    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:50:05.148159 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:50:05.148114    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:50:20.003323 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:20.003287    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:50:35.149096 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:50:35.149009    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:50:45.790779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:45.790744    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:50:51.123333 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:51.123253    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"]
Nov 01 15:50:51.229087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:51.229059    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"]
Nov 01 15:50:56.123639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:56.123580    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97" podUID="4982e142-162e-41bd-8029-ce8196ec44cc" containerName="payload-processing" containerID="cri-o://b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934" gracePeriod=25
Nov 01 15:50:56.230083 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:56.230039    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5" podUID="644dc96f-9456-4358-b65e-30a3ee01e354" containerName="payload-pre-processing" containerID="cri-o://1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5" gracePeriod=25
Nov 01 15:50:56.433642 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:56.433535    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"]
Nov 01 15:50:56.433862 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:56.433753    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft" podUID="85f84d1a-1a42-4e4a-aab5-a015f00420d3" containerName="maas-api" containerID="cri-o://dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278" gracePeriod=30
Nov 01 15:50:57.669480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.669455    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:50:57.772297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.772204    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-metrics-certs\") pod \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") "
Nov 01 15:50:57.772297 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.772291    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls\") pod \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") "
Nov 01 15:50:57.772573 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.772320    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f84d1a-1a42-4e4a-aab5-a015f00420d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f84d1a-1a42-4e4a-aab5-a015f00420d3-openshift-service-ca\") pod \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") "
Nov 01 15:50:57.772573 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.772352    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f84d1a-1a42-4e4a-aab5-a015f00420d3-kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/85f84d1a-1a42-4e4a-aab5-a015f00420d3-kube-api-access-p9chl\") pod \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\" (UID: \"85f84d1a-1a42-4e4a-aab5-a015f00420d3\") "
Nov 01 15:50:57.772769 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.772733    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f84d1a-1a42-4e4a-aab5-a015f00420d3-openshift-service-ca" pod "85f84d1a-1a42-4e4a-aab5-a015f00420d3" (UID: "85f84d1a-1a42-4e4a-aab5-a015f00420d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:50:57.774379 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.774354    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls" pod "85f84d1a-1a42-4e4a-aab5-a015f00420d3" (UID: "85f84d1a-1a42-4e4a-aab5-a015f00420d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:50:57.774379 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.774369    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-metrics-certs" pod "85f84d1a-1a42-4e4a-aab5-a015f00420d3" (UID: "85f84d1a-1a42-4e4a-aab5-a015f00420d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:50:57.774530 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.774510    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f84d1a-1a42-4e4a-aab5-a015f00420d3-kube-api-access-p9chl" pod "85f84d1a-1a42-4e4a-aab5-a015f00420d3" (UID: "85f84d1a-1a42-4e4a-aab5-a015f00420d3"). InnerVolumeSpecName "kube-api-access-p9chl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:50:57.873929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.873896    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9chl\" (UniqueName: \"kubernetes.io/projected/85f84d1a-1a42-4e4a-aab5-a015f00420d3-kube-api-access-p9chl\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:50:57.873929 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.873925    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:50:57.874127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.873936    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f84d1a-1a42-4e4a-aab5-a015f00420d3-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:50:57.874127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.873947    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f84d1a-1a42-4e4a-aab5-a015f00420d3-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:50:57.928782 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.928734    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5" podUID="644dc96f-9456-4358-b65e-30a3ee01e354" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:50:57.928976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:57.928737    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97" podUID="4982e142-162e-41bd-8029-ce8196ec44cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:50:58.403132 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.403099    5423 generic.go:356] "Generic (PLEG): container finished" podID="85f84d1a-1a42-4e4a-aab5-a015f00420d3" containerID="dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278" exitCode=0
Nov 01 15:50:58.403317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.403148    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"
Nov 01 15:50:58.403317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.403183    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft" event={"ID":"85f84d1a-1a42-4e4a-aab5-a015f00420d3","Type":"ContainerDied","Data":"dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278"}
Nov 01 15:50:58.403317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.403241    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft" event={"ID":"85f84d1a-1a42-4e4a-aab5-a015f00420d3","Type":"ContainerDied","Data":"4d2ceb5f6635e44e5d7fac9014457dfbf1af8a3764481a2d26028f52967b39fe"}
Nov 01 15:50:58.403317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.403265    5423 scope.go:122] "RemoveContainer" containerID="dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278"
Nov 01 15:50:58.412231 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.412215    5423 scope.go:122] "RemoveContainer" containerID="dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278"
Nov 01 15:50:58.412526 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:50:58.412505    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278\": container with ID starting with dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278 not found: ID does not exist" containerID="dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278"
Nov 01 15:50:58.412583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.412534    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278"} err="failed to get container status \"dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278\": rpc error: code = NotFound desc = could not find container \"dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278\": container with ID starting with dc90b062fccb646ed30acd82873ce8a64a6ab867f9d7df4f63b6a69a2f0f7278 not found: ID does not exist"
Nov 01 15:50:58.569376 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.569340    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"]
Nov 01 15:50:58.620639 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:58.620602    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9c2ba6-56df6d8c97-rsfft"]
Nov 01 15:50:59.959304 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:50:59.959265    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f84d1a-1a42-4e4a-aab5-a015f00420d3" path="/var/lib/kubelet/pods/85f84d1a-1a42-4e4a-aab5-a015f00420d3/volumes"
Nov 01 15:51:00.924332 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:00.924287    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:51:05.147478 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:51:05.147435    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:51:07.928235 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:07.928188    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97" podUID="4982e142-162e-41bd-8029-ce8196ec44cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:51:07.928691 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:07.928198    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5" podUID="644dc96f-9456-4358-b65e-30a3ee01e354" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:51:16.372858 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.372826    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:51:16.449100 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.449069    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4982e142-162e-41bd-8029-ce8196ec44cc-kube-api-access-grnmr\" (UniqueName: \"kubernetes.io/projected/4982e142-162e-41bd-8029-ce8196ec44cc-kube-api-access-grnmr\") pod \"4982e142-162e-41bd-8029-ce8196ec44cc\" (UID: \"4982e142-162e-41bd-8029-ce8196ec44cc\") "
Nov 01 15:51:16.449264 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.449116    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4982e142-162e-41bd-8029-ce8196ec44cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982e142-162e-41bd-8029-ce8196ec44cc-plugins-config-volume\") pod \"4982e142-162e-41bd-8029-ce8196ec44cc\" (UID: \"4982e142-162e-41bd-8029-ce8196ec44cc\") "
Nov 01 15:51:16.449538 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.449512    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4982e142-162e-41bd-8029-ce8196ec44cc-plugins-config-volume" pod "4982e142-162e-41bd-8029-ce8196ec44cc" (UID: "4982e142-162e-41bd-8029-ce8196ec44cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:51:16.451605 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.451579    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4982e142-162e-41bd-8029-ce8196ec44cc-kube-api-access-grnmr" pod "4982e142-162e-41bd-8029-ce8196ec44cc" (UID: "4982e142-162e-41bd-8029-ce8196ec44cc"). InnerVolumeSpecName "kube-api-access-grnmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:51:16.470517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.470496    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:51:16.495157 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.495134    5423 generic.go:356] "Generic (PLEG): container finished" podID="644dc96f-9456-4358-b65e-30a3ee01e354" containerID="1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5" exitCode=0
Nov 01 15:51:16.495270 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.495223    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5" event={"ID":"644dc96f-9456-4358-b65e-30a3ee01e354","Type":"ContainerDied","Data":"1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5"}
Nov 01 15:51:16.495347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.495304    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5" event={"ID":"644dc96f-9456-4358-b65e-30a3ee01e354","Type":"ContainerDied","Data":"d996f8023833a83e16ef865599dcd6d8b4db24f2a9b71782b2d3cbc89cdff475"}
Nov 01 15:51:16.495347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.495335    5423 scope.go:122] "RemoveContainer" containerID="1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5"
Nov 01 15:51:16.495517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.495501    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"
Nov 01 15:51:16.497347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.497321    5423 generic.go:356] "Generic (PLEG): container finished" podID="4982e142-162e-41bd-8029-ce8196ec44cc" containerID="b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934" exitCode=0
Nov 01 15:51:16.497492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.497357    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"
Nov 01 15:51:16.497492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.497410    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97" event={"ID":"4982e142-162e-41bd-8029-ce8196ec44cc","Type":"ContainerDied","Data":"b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934"}
Nov 01 15:51:16.497492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.497460    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97" event={"ID":"4982e142-162e-41bd-8029-ce8196ec44cc","Type":"ContainerDied","Data":"f7e52c32b07f877e6a8010484d754ffcd3855f1f143a0a5adc9baa5fe75048a6"}
Nov 01 15:51:16.504370 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.504356    5423 scope.go:122] "RemoveContainer" containerID="1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5"
Nov 01 15:51:16.504690 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:51:16.504674    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5\": container with ID starting with 1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5 not found: ID does not exist" containerID="1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5"
Nov 01 15:51:16.504779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.504700    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5"} err="failed to get container status \"1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5\": rpc error: code = NotFound desc = could not find container \"1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5\": container with ID starting with 1e882112d2b2bc19bf8497fa2d691cfc1193abc0ce626781f0a7e24e548018e5 not found: ID does not exist"
Nov 01 15:51:16.504779 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.504723    5423 scope.go:122] "RemoveContainer" containerID="b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934"
Nov 01 15:51:16.512439 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.512414    5423 scope.go:122] "RemoveContainer" containerID="b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934"
Nov 01 15:51:16.512683 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:51:16.512665    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934\": container with ID starting with b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934 not found: ID does not exist" containerID="b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934"
Nov 01 15:51:16.512770 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.512687    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934"} err="failed to get container status \"b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934\": rpc error: code = NotFound desc = could not find container \"b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934\": container with ID starting with b7517962b30adbac07e6f8d65f87dd3928f1689a805759434fb2555267f0b934 not found: ID does not exist"
Nov 01 15:51:16.550683 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.550660    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644dc96f-9456-4358-b65e-30a3ee01e354-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644dc96f-9456-4358-b65e-30a3ee01e354-plugins-config-volume\") pod \"644dc96f-9456-4358-b65e-30a3ee01e354\" (UID: \"644dc96f-9456-4358-b65e-30a3ee01e354\") "
Nov 01 15:51:16.550787 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.550725    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644dc96f-9456-4358-b65e-30a3ee01e354-kube-api-access-z5xfb\" (UniqueName: \"kubernetes.io/projected/644dc96f-9456-4358-b65e-30a3ee01e354-kube-api-access-z5xfb\") pod \"644dc96f-9456-4358-b65e-30a3ee01e354\" (UID: \"644dc96f-9456-4358-b65e-30a3ee01e354\") "
Nov 01 15:51:16.550920 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.550906    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grnmr\" (UniqueName: \"kubernetes.io/projected/4982e142-162e-41bd-8029-ce8196ec44cc-kube-api-access-grnmr\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:16.550989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.550920    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982e142-162e-41bd-8029-ce8196ec44cc-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:16.551079 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.551042    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644dc96f-9456-4358-b65e-30a3ee01e354-plugins-config-volume" pod "644dc96f-9456-4358-b65e-30a3ee01e354" (UID: "644dc96f-9456-4358-b65e-30a3ee01e354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:51:16.552601 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.552583    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644dc96f-9456-4358-b65e-30a3ee01e354-kube-api-access-z5xfb" pod "644dc96f-9456-4358-b65e-30a3ee01e354" (UID: "644dc96f-9456-4358-b65e-30a3ee01e354"). InnerVolumeSpecName "kube-api-access-z5xfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:51:16.651863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.651808    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644dc96f-9456-4358-b65e-30a3ee01e354-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:16.651863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.651830    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5xfb\" (UniqueName: \"kubernetes.io/projected/644dc96f-9456-4358-b65e-30a3ee01e354-kube-api-access-z5xfb\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:16.714782 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.714757    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"]
Nov 01 15:51:16.718294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.718273    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9c2ba6-d65599b89dvw97"]
Nov 01 15:51:16.916194 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.916127    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"]
Nov 01 15:51:16.917975 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:16.917954    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9c2ba6-54555rvds5"]
Nov 01 15:51:17.959842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:17.959810    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4982e142-162e-41bd-8029-ce8196ec44cc" path="/var/lib/kubelet/pods/4982e142-162e-41bd-8029-ce8196ec44cc/volumes"
Nov 01 15:51:17.960367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:17.960348    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644dc96f-9456-4358-b65e-30a3ee01e354" path="/var/lib/kubelet/pods/644dc96f-9456-4358-b65e-30a3ee01e354/volumes"
Nov 01 15:51:30.084367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:30.084335    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"]
Nov 01 15:51:30.084930 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:30.084578    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy" containerID="cri-o://d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d" gracePeriod=30
Nov 01 15:51:35.147864 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:51:35.147827    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:51:35.335626 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.335603    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:51:35.442988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.442922    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-credential-socket\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.442988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.442957    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845f9010-0051-4025-88b2-8269895a692d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845f9010-0051-4025-88b2-8269895a692d-istiod-ca-cert\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.442988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.442987    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-socket\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443161    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-istio-token\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-istio-token\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443199    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-kube-api-access-7r588\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-kube-api-access-7r588\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443248    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-credential-socket" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443267    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-envoy\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443287    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-certs\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443263    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-socket" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443333    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-data\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443341    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845f9010-0051-4025-88b2-8269895a692d-istiod-ca-cert" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:51:35.443432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443426    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/845f9010-0051-4025-88b2-8269895a692d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845f9010-0051-4025-88b2-8269895a692d-istio-podinfo\") pod \"845f9010-0051-4025-88b2-8269895a692d\" (UID: \"845f9010-0051-4025-88b2-8269895a692d\") "
Nov 01 15:51:35.443952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443699    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-certs" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:51:35.443952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443769    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.443952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443791    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845f9010-0051-4025-88b2-8269895a692d-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.443952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443806    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.443952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.443875    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-data" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:51:35.445528 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.445500    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-envoy" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:51:35.445528 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.445514    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/845f9010-0051-4025-88b2-8269895a692d-istio-podinfo" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:51:35.445815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.445792    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-istio-token" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:51:35.445912 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.445811    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-kube-api-access-7r588" pod "845f9010-0051-4025-88b2-8269895a692d" (UID: "845f9010-0051-4025-88b2-8269895a692d"). InnerVolumeSpecName "kube-api-access-7r588". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:51:35.544540 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.544512    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.544540 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.544536    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r588\" (UniqueName: \"kubernetes.io/projected/845f9010-0051-4025-88b2-8269895a692d-kube-api-access-7r588\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.544700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.544551    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.544700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.544562    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.544700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.544570    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845f9010-0051-4025-88b2-8269895a692d-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.544700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.544577    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845f9010-0051-4025-88b2-8269895a692d-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:51:35.592666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.592642    5423 generic.go:356] "Generic (PLEG): container finished" podID="845f9010-0051-4025-88b2-8269895a692d" containerID="d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d" exitCode=0
Nov 01 15:51:35.592795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.592697    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"
Nov 01 15:51:35.592795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.592723    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" event={"ID":"845f9010-0051-4025-88b2-8269895a692d","Type":"ContainerDied","Data":"d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d"}
Nov 01 15:51:35.592795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.592777    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5" event={"ID":"845f9010-0051-4025-88b2-8269895a692d","Type":"ContainerDied","Data":"06acca8bdcc37ca6654b229e4e44c95d05e21c40ecf45b232961023460d1cd7c"}
Nov 01 15:51:35.592957 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.592799    5423 scope.go:122] "RemoveContainer" containerID="d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d"
Nov 01 15:51:35.601325 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.601312    5423 scope.go:122] "RemoveContainer" containerID="7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989"
Nov 01 15:51:35.608909 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.608892    5423 scope.go:122] "RemoveContainer" containerID="d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d"
Nov 01 15:51:35.609147 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:51:35.609128    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d\": container with ID starting with d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d not found: ID does not exist" containerID="d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d"
Nov 01 15:51:35.609186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.609156    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d"} err="failed to get container status \"d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d\": rpc error: code = NotFound desc = could not find container \"d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d\": container with ID starting with d36e58f01c2c97f75a8999e7b38ebf5fdf77176133401410c8386f78eb8d480d not found: ID does not exist"
Nov 01 15:51:35.609186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.609172    5423 scope.go:122] "RemoveContainer" containerID="7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989"
Nov 01 15:51:35.609492 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:51:35.609474    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989\": container with ID starting with 7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989 not found: ID does not exist" containerID="7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989"
Nov 01 15:51:35.609535 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.609498    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989"} err="failed to get container status \"7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989\": rpc error: code = NotFound desc = could not find container \"7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989\": container with ID starting with 7829720c6211fe5c6655568feb68ed50e35e5522de4efdf73b0e6dabfc62c989 not found: ID does not exist"
Nov 01 15:51:35.760052 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.759988    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"]
Nov 01 15:51:35.763295 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.763275    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9c2ba6-openshift-default-74d95d57-5wwl5"]
Nov 01 15:51:35.965245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:35.965213    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845f9010-0051-4025-88b2-8269895a692d" path="/var/lib/kubelet/pods/845f9010-0051-4025-88b2-8269895a692d/volumes"
Nov 01 15:51:50.368037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:51:50.367997    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:52:05.148427 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:52:05.148372    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:52:15.440900 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.440865    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"]
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441232    5423 cpu_manager.go:414] "containerMap: removing container" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441242    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441257    5423 cpu_manager.go:414] "containerMap: removing container" podUID="85f84d1a-1a42-4e4a-aab5-a015f00420d3" containerName="maas-api"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441261    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f84d1a-1a42-4e4a-aab5-a015f00420d3" containerName="maas-api"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441270    5423 cpu_manager.go:414] "containerMap: removing container" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441276    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441291    5423 cpu_manager.go:414] "containerMap: removing container" podUID="4982e142-162e-41bd-8029-ce8196ec44cc" containerName="payload-processing"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441296    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4982e142-162e-41bd-8029-ce8196ec44cc" containerName="payload-processing"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441306    5423 cpu_manager.go:414] "containerMap: removing container" podUID="644dc96f-9456-4358-b65e-30a3ee01e354" containerName="payload-pre-processing"
Nov 01 15:52:15.441352 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441311    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644dc96f-9456-4358-b65e-30a3ee01e354" containerName="payload-pre-processing"
Nov 01 15:52:15.441705 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441365    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="4982e142-162e-41bd-8029-ce8196ec44cc" containerName="payload-processing"
Nov 01 15:52:15.441705 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441374    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="644dc96f-9456-4358-b65e-30a3ee01e354" containerName="payload-pre-processing"
Nov 01 15:52:15.441705 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441397    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy"
Nov 01 15:52:15.441705 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441404    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f84d1a-1a42-4e4a-aab5-a015f00420d3" containerName="maas-api"
Nov 01 15:52:15.441705 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.441411    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="845f9010-0051-4025-88b2-8269895a692d" containerName="istio-proxy"
Nov 01 15:52:15.444087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.444072    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.546688 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.546653    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-model-cache\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.546860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.546704    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tmp-dir\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.546860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.546791    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-dshm\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.546860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.546826    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qdnr\" (UniqueName: \"kubernetes.io/projected/a431f9c4-28e9-45c5-97fb-37ba5bf54248-kube-api-access-4qdnr\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.547034 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.546906    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-home\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.547034 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.546948    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tls-certs\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.550315 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.550301    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-835e7bd2-kserve-self-signed-certs\""
Nov 01 15:52:15.551749 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.551732    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"]
Nov 01 15:52:15.648227 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648194    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-model-cache\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.648435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648243    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tmp-dir\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.648435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648289    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-dshm\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.648435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648317    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdnr\" (UniqueName: \"kubernetes.io/projected/a431f9c4-28e9-45c5-97fb-37ba5bf54248-kube-api-access-4qdnr\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.648435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648372    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-home\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.648435 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648429    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tls-certs\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.648733 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648702    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-model-cache\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.648889 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648746    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tmp-dir\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.648889 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.648862    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-home\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.650595 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.650578    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-dshm\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.650849 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.650830    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tls-certs\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.705644 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.705576    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qdnr\" (UniqueName: \"kubernetes.io/projected/a431f9c4-28e9-45c5-97fb-37ba5bf54248-kube-api-access-4qdnr\") pod \"e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") " pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:15.756107 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:15.756078    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:16.143463 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:16.143438    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"]
Nov 01 15:52:16.145922 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:52:16.145899    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda431f9c4_28e9_45c5_97fb_37ba5bf54248.slice/crio-be026452e69bd8d65e03edc74fb6eff4dc9c352e7fe0f2c6b8f7251f69ce1a5f WatchSource:0}: Error finding container be026452e69bd8d65e03edc74fb6eff4dc9c352e7fe0f2c6b8f7251f69ce1a5f: Status 404 returned error can't find the container with id be026452e69bd8d65e03edc74fb6eff4dc9c352e7fe0f2c6b8f7251f69ce1a5f
Nov 01 15:52:16.779299 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:16.779266    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx" event={"ID":"a431f9c4-28e9-45c5-97fb-37ba5bf54248","Type":"ContainerStarted","Data":"431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6"}
Nov 01 15:52:16.779794 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:16.779305    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx" event={"ID":"a431f9c4-28e9-45c5-97fb-37ba5bf54248","Type":"ContainerStarted","Data":"be026452e69bd8d65e03edc74fb6eff4dc9c352e7fe0f2c6b8f7251f69ce1a5f"}
Nov 01 15:52:16.894855 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:16.894807    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx" podStartSLOduration=1.894793375 podStartE2EDuration="1.894793375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:16.89405666 +0000 UTC m=+5677.059541957" watchObservedRunningTime="2026-11-01 15:52:16.894793375 +0000 UTC m=+5677.060278672"
Nov 01 15:52:17.756653 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:17.756620    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:19.905108 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:19.905037    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:52:25.757048 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:25.757017    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:25.769207 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:25.769183    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:25.866188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:25.866160    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:52:35.150208 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:52:35.150161    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:52:39.295986 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:39.295942    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:52:42.063966 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:42.063915    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:52:42.064539 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:42.063991    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:52:43.246089 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:43.246064    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:52:43.263458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:43.263438    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:52:53.575589 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:53.575554    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"]
Nov 01 15:52:54.476848 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:54.476817    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:52:57.061411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:57.060526    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:52:57.061411 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:52:57.060618    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:53:04.151638 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:04.151593    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:53:05.148871 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:53:05.148832    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:53:08.595035 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.594973    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx" podUID="a431f9c4-28e9-45c5-97fb-37ba5bf54248" containerName="main" containerID="cri-o://431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6" gracePeriod=45
Nov 01 15:53:08.842198 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.842172    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:53:08.910129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910071    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tmp-dir\") pod \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") "
Nov 01 15:53:08.910129 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910108    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-model-cache\") pod \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") "
Nov 01 15:53:08.910309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910132    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tls-certs\" (UniqueName: \"kubernetes.io/secret/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tls-certs\") pod \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") "
Nov 01 15:53:08.910309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910157    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a431f9c4-28e9-45c5-97fb-37ba5bf54248-kube-api-access-4qdnr\" (UniqueName: \"kubernetes.io/projected/a431f9c4-28e9-45c5-97fb-37ba5bf54248-kube-api-access-4qdnr\") pod \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") "
Nov 01 15:53:08.910309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910198    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-home\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-home\") pod \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") "
Nov 01 15:53:08.910309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910251    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-dshm\") pod \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\" (UID: \"a431f9c4-28e9-45c5-97fb-37ba5bf54248\") "
Nov 01 15:53:08.910562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910364    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tmp-dir" pod "a431f9c4-28e9-45c5-97fb-37ba5bf54248" (UID: "a431f9c4-28e9-45c5-97fb-37ba5bf54248"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:53:08.910562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910378    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-model-cache" pod "a431f9c4-28e9-45c5-97fb-37ba5bf54248" (UID: "a431f9c4-28e9-45c5-97fb-37ba5bf54248"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:53:08.910562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910501    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-home" pod "a431f9c4-28e9-45c5-97fb-37ba5bf54248" (UID: "a431f9c4-28e9-45c5-97fb-37ba5bf54248"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:53:08.910717 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910606    5423 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tmp-dir\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:53:08.910717 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910620    5423 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-model-cache\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:53:08.910717 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.910630    5423 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-home\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:53:08.912309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.912281    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-dshm" pod "a431f9c4-28e9-45c5-97fb-37ba5bf54248" (UID: "a431f9c4-28e9-45c5-97fb-37ba5bf54248"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:53:08.912438 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.912282    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tls-certs" pod "a431f9c4-28e9-45c5-97fb-37ba5bf54248" (UID: "a431f9c4-28e9-45c5-97fb-37ba5bf54248"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:53:08.912438 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:08.912359    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a431f9c4-28e9-45c5-97fb-37ba5bf54248-kube-api-access-4qdnr" pod "a431f9c4-28e9-45c5-97fb-37ba5bf54248" (UID: "a431f9c4-28e9-45c5-97fb-37ba5bf54248"). InnerVolumeSpecName "kube-api-access-4qdnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:53:09.011282 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.011254    5423 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a431f9c4-28e9-45c5-97fb-37ba5bf54248-dshm\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:53:09.011282 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.011281    5423 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a431f9c4-28e9-45c5-97fb-37ba5bf54248-tls-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:53:09.011498 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.011291    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qdnr\" (UniqueName: \"kubernetes.io/projected/a431f9c4-28e9-45c5-97fb-37ba5bf54248-kube-api-access-4qdnr\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:53:09.102117 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.102081    5423 generic.go:356] "Generic (PLEG): container finished" podID="a431f9c4-28e9-45c5-97fb-37ba5bf54248" containerID="431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6" exitCode=0
Nov 01 15:53:09.102285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.102130    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"
Nov 01 15:53:09.102285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.102152    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx" event={"ID":"a431f9c4-28e9-45c5-97fb-37ba5bf54248","Type":"ContainerDied","Data":"431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6"}
Nov 01 15:53:09.102285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.102183    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx" event={"ID":"a431f9c4-28e9-45c5-97fb-37ba5bf54248","Type":"ContainerDied","Data":"be026452e69bd8d65e03edc74fb6eff4dc9c352e7fe0f2c6b8f7251f69ce1a5f"}
Nov 01 15:53:09.102285 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.102197    5423 scope.go:122] "RemoveContainer" containerID="431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6"
Nov 01 15:53:09.111668 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.111651    5423 scope.go:122] "RemoveContainer" containerID="431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6"
Nov 01 15:53:09.111941 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:53:09.111906    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6\": container with ID starting with 431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6 not found: ID does not exist" containerID="431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6"
Nov 01 15:53:09.111995 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.111947    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6"} err="failed to get container status \"431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6\": rpc error: code = NotFound desc = could not find container \"431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6\": container with ID starting with 431ea8be4039bfdd3690ba75011cabe9612eb5786bd570b5f4a393f9728cf3f6 not found: ID does not exist"
Nov 01 15:53:09.268727 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.268650    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"]
Nov 01 15:53:09.273239 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.273217    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-835e7bd2-kserve-6c8984b9f7-9ljmx"]
Nov 01 15:53:09.958914 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:09.958882    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a431f9c4-28e9-45c5-97fb-37ba5bf54248" path="/var/lib/kubelet/pods/a431f9c4-28e9-45c5-97fb-37ba5bf54248/volumes"
Nov 01 15:53:12.047448 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:12.047420    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:53:21.262144 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:21.262106    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:53:35.147792 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:53:35.147750    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:53:35.471821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.471743    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"]
Nov 01 15:53:35.472255 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.472237    5423 cpu_manager.go:414] "containerMap: removing container" podUID="a431f9c4-28e9-45c5-97fb-37ba5bf54248" containerName="main"
Nov 01 15:53:35.472319 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.472257    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a431f9c4-28e9-45c5-97fb-37ba5bf54248" containerName="main"
Nov 01 15:53:35.472369 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.472360    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="a431f9c4-28e9-45c5-97fb-37ba5bf54248" containerName="main"
Nov 01 15:53:35.475379 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.475365    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.544938 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.544910    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-dual-3436ba-openshift-default-dockercfg-hdq2b\""
Nov 01 15:53:35.582166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.582132    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"]
Nov 01 15:53:35.589870 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.589837    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.590004 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.589892    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.590075 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.590049    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.590121 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.590096    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.590188 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.590168    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.590224 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.590200    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.590271 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.590226    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.590381 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.590361    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.590469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.590452    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcjs\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-kube-api-access-4pcjs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.642224 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.642194    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:53:35.691294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691261    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcjs\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-kube-api-access-4pcjs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.691480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691336    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.691480 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691360    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.691574 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691476    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.691574 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691523    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.691574 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691554    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.691737 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691593    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.691737 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691624    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.691860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.691757    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.692023 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.692000    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.692123 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.692096    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.692187 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.692122    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.692309 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.692285    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.692632 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.692609    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.693956 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.693933    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.694094 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.694076    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.759336 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.759251    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcjs\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-kube-api-access-4pcjs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.759921 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.759867    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-hjksq\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:35.803228 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:35.803196    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:36.228815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:36.228784    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"]
Nov 01 15:53:36.231190 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:53:36.231163    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1328aa87_1471_4bf0_8304_6fbe7e236b7e.slice/crio-f165251565d21f17368ed90647d1a7d871cfe27741e170ec6323e3b61f2e75f8 WatchSource:0}: Error finding container f165251565d21f17368ed90647d1a7d871cfe27741e170ec6323e3b61f2e75f8: Status 404 returned error can't find the container with id f165251565d21f17368ed90647d1a7d871cfe27741e170ec6323e3b61f2e75f8
Nov 01 15:53:36.233328 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:36.233298    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:53:36.233402 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:36.233359    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:53:36.233450 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:36.233404    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:53:36.249735 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:36.249712    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" event={"ID":"1328aa87-1471-4bf0-8304-6fbe7e236b7e","Type":"ContainerStarted","Data":"f165251565d21f17368ed90647d1a7d871cfe27741e170ec6323e3b61f2e75f8"}
Nov 01 15:53:37.255200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:37.255164    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" event={"ID":"1328aa87-1471-4bf0-8304-6fbe7e236b7e","Type":"ContainerStarted","Data":"3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe"}
Nov 01 15:53:37.389713 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:37.389657    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" podStartSLOduration=2.389643214 podStartE2EDuration="2.389643214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:37.388108528 +0000 UTC m=+5757.553593825" watchObservedRunningTime="2026-11-01 15:53:37.389643214 +0000 UTC m=+5757.555128511"
Nov 01 15:53:37.803750 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:37.803712    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:37.808669 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:37.808642    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:38.259522 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:38.259492    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:38.260335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:38.260317    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:53:45.958797 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:45.958767    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"]
Nov 01 15:53:45.963802 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:45.963783    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.014424 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.014375    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"]
Nov 01 15:53:46.074057 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.074025    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-dual-3436ba-serving-cert\""
Nov 01 15:53:46.092202 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.092166    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7254e30-e914-477c-a827-dc651fbd374d-openshift-service-ca\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.092404 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.092296    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqqv\" (UniqueName: \"kubernetes.io/projected/b7254e30-e914-477c-a827-dc651fbd374d-kube-api-access-sgqqv\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.092404 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.092334    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-metrics-certs\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.092552 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.092533    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.194920 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.194880    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.195182 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.195119    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7254e30-e914-477c-a827-dc651fbd374d-openshift-service-ca\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.195182 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.195177    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqqv\" (UniqueName: \"kubernetes.io/projected/b7254e30-e914-477c-a827-dc651fbd374d-kube-api-access-sgqqv\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.195289 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.195201    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-metrics-certs\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.195445 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:53:46.195418    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-serving-cert: secret "maas-api-e2e-dual-3436ba-serving-cert" not found
Nov 01 15:53:46.195571 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:53:46.195507    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls podName:b7254e30-e914-477c-a827-dc651fbd374d nodeName:}" failed. No retries permitted until 2026-11-01 15:53:46.695482912 +0000 UTC m=+5766.860968195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls") pod "maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m" (UID: "b7254e30-e914-477c-a827-dc651fbd374d") : secret "maas-api-e2e-dual-3436ba-serving-cert" not found
Nov 01 15:53:46.196334 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.196307    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7254e30-e914-477c-a827-dc651fbd374d-openshift-service-ca\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.202534 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.198903    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-metrics-certs\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.253664 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.253603    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqqv\" (UniqueName: \"kubernetes.io/projected/b7254e30-e914-477c-a827-dc651fbd374d-kube-api-access-sgqqv\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.700796 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.700753    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.703119 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.703088    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls\") pod \"maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") " pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:46.876588 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:46.876559    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:47.245286 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:47.245262    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"]
Nov 01 15:53:47.251536 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:53:47.251511    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7254e30_e914_477c_a827_dc651fbd374d.slice/crio-30195e83d9e9738a9f95cbc14bb8547c74ad65efedadb5150ac361a392b259e6 WatchSource:0}: Error finding container 30195e83d9e9738a9f95cbc14bb8547c74ad65efedadb5150ac361a392b259e6: Status 404 returned error can't find the container with id 30195e83d9e9738a9f95cbc14bb8547c74ad65efedadb5150ac361a392b259e6
Nov 01 15:53:47.299859 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:47.299827    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m" event={"ID":"b7254e30-e914-477c-a827-dc651fbd374d","Type":"ContainerStarted","Data":"30195e83d9e9738a9f95cbc14bb8547c74ad65efedadb5150ac361a392b259e6"}
Nov 01 15:53:48.306518 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:48.306487    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m" event={"ID":"b7254e30-e914-477c-a827-dc651fbd374d","Type":"ContainerStarted","Data":"776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4"}
Nov 01 15:53:48.306940 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:48.306606    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:48.443149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:48.443074    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m" podStartSLOduration=2.476282146 podStartE2EDuration="3.443039702s" totalImagesPullingTime="966.757556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:48.442570967 +0000 UTC m=+5768.608056264" watchObservedRunningTime="2026-11-01 15:53:48.443039702 +0000 UTC m=+5768.608525001"
Nov 01 15:53:54.316361 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:54.316279    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:53:56.538187 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:53:56.538151    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:54:05.147702 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:54:05.147658    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:54:05.811017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:05.810974    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"]
Nov 01 15:54:05.816775 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:05.816748    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012191 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012154    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012400 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012207    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012400 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012233    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012400 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012291    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012400 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012337    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012400 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012365    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012435    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012479    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdd7k\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-kube-api-access-jdd7k\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.012670 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.012505    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.014013 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.013990    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"]
Nov 01 15:54:06.113667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113578    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.113667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113622    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.113667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113652    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113695    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113770    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113817    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113866    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113910    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdd7k\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-kube-api-access-jdd7k\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.113937    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.114004    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114335 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.114233    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.114449    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.114545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.114498    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.115127 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.115103    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.116209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.116188    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.116987 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.116967    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.171850 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.171779    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.172294 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.171861    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdd7k\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-kube-api-access-jdd7k\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.397047 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.396969    5423 generic.go:356] "Generic (PLEG): container finished" podID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerID="3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe" exitCode=0
Nov 01 15:54:06.397220 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.397045    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" event={"ID":"1328aa87-1471-4bf0-8304-6fbe7e236b7e","Type":"ContainerDied","Data":"3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe"}
Nov 01 15:54:06.397502 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.397486    5423 scope.go:122] "RemoveContainer" containerID="3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe"
Nov 01 15:54:06.398724 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.398699    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:06.398805 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.398765    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:06.398843 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.398810    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:06.433675 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.433658    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:06.859500 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.859471    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"]
Nov 01 15:54:06.861989 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:54:06.861950    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ec076b_ab68_4533_83c5_95f4b980b9fc.slice/crio-7f9a436dee5938e281afe8c6123f0a27c6e275551a30ec900c7f938fc73596d2 WatchSource:0}: Error finding container 7f9a436dee5938e281afe8c6123f0a27c6e275551a30ec900c7f938fc73596d2: Status 404 returned error can't find the container with id 7f9a436dee5938e281afe8c6123f0a27c6e275551a30ec900c7f938fc73596d2
Nov 01 15:54:06.864274 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.864237    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:06.864423 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.864320    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:06.864423 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:06.864348    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:07.420852 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.420804    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" event={"ID":"1328aa87-1471-4bf0-8304-6fbe7e236b7e","Type":"ContainerStarted","Data":"255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7"}
Nov 01 15:54:07.423410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.423310    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" event={"ID":"e5ec076b-ab68-4533-83c5-95f4b980b9fc","Type":"ContainerStarted","Data":"b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc"}
Nov 01 15:54:07.423410 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.423361    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" event={"ID":"e5ec076b-ab68-4533-83c5-95f4b980b9fc","Type":"ContainerStarted","Data":"7f9a436dee5938e281afe8c6123f0a27c6e275551a30ec900c7f938fc73596d2"}
Nov 01 15:54:07.435173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.435129    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:07.435976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.435890    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Nov 01 15:54:07.435976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.435944    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Nov 01 15:54:07.687158 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.687034    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podStartSLOduration=2.687021337 podStartE2EDuration="2.687021337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:07.685680689 +0000 UTC m=+5787.851165986" watchObservedRunningTime="2026-11-01 15:54:07.687021337 +0000 UTC m=+5787.852506628"
Nov 01 15:54:07.809892 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.806766    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:54:07.835409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.827409    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 01 15:54:07.835409 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:07.827474    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 01 15:54:08.260456 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:08.260421    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:54:08.439546 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:08.434215    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Nov 01 15:54:08.439546 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:08.434274    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Nov 01 15:54:08.805959 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:08.805518    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 01 15:54:08.805959 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:08.805577    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 01 15:54:09.434028 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:09.433989    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Nov 01 15:54:09.434469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:09.434438    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Nov 01 15:54:09.812469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:09.806751    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 01 15:54:09.812469 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:09.806811    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 01 15:54:10.435814 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:10.435772    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Nov 01 15:54:10.436095 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:10.436065    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Nov 01 15:54:10.806497 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:10.806466    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 01 15:54:10.806828 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:10.806796    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 01 15:54:11.436863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:11.436818    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Nov 01 15:54:11.438871 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:11.438834    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Nov 01 15:54:11.810843 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:11.810810    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 01 15:54:11.811171 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:11.811141    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 01 15:54:12.434683 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:12.434647    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Nov 01 15:54:12.434869 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:12.434714    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Nov 01 15:54:12.813272 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:12.813234    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:54:12.814313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:12.814289    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:54:13.438373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:13.438344    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:13.438640 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:13.438611    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:13.439320 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:13.439299    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:54:35.148276 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:54:35.148220    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:54:36.050693 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.050665    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"]
Nov 01 15:54:36.054179 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.054161    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136257 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136228    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136279    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136339    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136372    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136463    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136567 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136546    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgg2\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-kube-api-access-mzgg2\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136572    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136647 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136598    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.136738 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.136724    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237407 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237350    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237415    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237442    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237480    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237533    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237617    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgg2\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-kube-api-access-mzgg2\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237644    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237683    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237776    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.237875 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237816    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.238416 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.237878    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.238485 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.238122    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.238545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.238491    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.238600 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.238541    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.240479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.240453    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.240609 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.240593    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.256922 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.256895    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"]
Nov 01 15:54:36.296354 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.296334    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.296658 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.296636    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgg2\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-kube-api-access-mzgg2\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-qgps5\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.367205 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.367174    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:36.793967 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.793922    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"]
Nov 01 15:54:36.796609 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:54:36.796582    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5384fc5_82e9_42b4_a0d6_d07469d9b048.slice/crio-7416c5c6257bff941688ddb0a960d4fe231f548c0c84a9f6b5a85070b4c8ee5e WatchSource:0}: Error finding container 7416c5c6257bff941688ddb0a960d4fe231f548c0c84a9f6b5a85070b4c8ee5e: Status 404 returned error can't find the container with id 7416c5c6257bff941688ddb0a960d4fe231f548c0c84a9f6b5a85070b4c8ee5e
Nov 01 15:54:36.798716 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.798687    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:36.798793 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.798750    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:36.798793 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:36.798780    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:37.584932 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:37.584891    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" event={"ID":"b5384fc5-82e9-42b4-a0d6-d07469d9b048","Type":"ContainerStarted","Data":"c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e"}
Nov 01 15:54:37.585345 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:37.584947    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" event={"ID":"b5384fc5-82e9-42b4-a0d6-d07469d9b048","Type":"ContainerStarted","Data":"7416c5c6257bff941688ddb0a960d4fe231f548c0c84a9f6b5a85070b4c8ee5e"}
Nov 01 15:54:37.755545 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:37.755493    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" podStartSLOduration=1.755479032 podStartE2EDuration="1.755479032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:37.753453174 +0000 UTC m=+5817.918938468" watchObservedRunningTime="2026-11-01 15:54:37.755479032 +0000 UTC m=+5817.920964330"
Nov 01 15:54:38.368209 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:38.368176    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:38.369704 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:38.369675    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 01 15:54:38.369858 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:38.369732    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 01 15:54:39.368008 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:39.367976    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 01 15:54:39.368454 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:39.368031    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 01 15:54:40.368110 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:40.368070    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 01 15:54:40.368638 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:40.368605    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 01 15:54:41.367604 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:41.367566    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 01 15:54:41.367813 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:41.367622    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 01 15:54:42.368358 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:42.368320    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Nov 01 15:54:42.368795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:42.368380    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Nov 01 15:54:43.372245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:43.372213    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:43.372690 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:43.372511    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:43.373229 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:43.373208    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:54:51.158262 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.158224    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"]
Nov 01 15:54:51.162968 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.162948    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207041 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207005    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207185 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207066    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac17e449-5699-4e32-8f9c-165ef0cbc298-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207185 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207095    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207195    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207242    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207310    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207433 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207400    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207474    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.207560 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.207519    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnm5\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-kube-api-access-krnm5\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.258440 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.258414    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"]
Nov 01 15:54:51.308562 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308533    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.308694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308576    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.308694 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308593    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.308821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308722    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.308821 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308809    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnm5\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-kube-api-access-krnm5\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.308934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308856    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.308934 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308887    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac17e449-5699-4e32-8f9c-165ef0cbc298-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.309045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308936    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.309045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308983    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.309045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.308887    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.309211 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.309092    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.309299 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.309275    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.309432 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.309416    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.309790 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.309764    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac17e449-5699-4e32-8f9c-165ef0cbc298-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.310754 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.310737    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.311142 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.311127    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.366475 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.366448    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.366661 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.366643    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnm5\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-kube-api-access-krnm5\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.475547 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.475480    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:51.898946 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.898923    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"]
Nov 01 15:54:51.900906 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:54:51.900879    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac17e449_5699_4e32_8f9c_165ef0cbc298.slice/crio-15cdcdcaa3ee019fa846835ecc22c071f814a40da269eab4db5525552b1f3e17 WatchSource:0}: Error finding container 15cdcdcaa3ee019fa846835ecc22c071f814a40da269eab4db5525552b1f3e17: Status 404 returned error can't find the container with id 15cdcdcaa3ee019fa846835ecc22c071f814a40da269eab4db5525552b1f3e17
Nov 01 15:54:51.903087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.903034    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:51.903200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.903130    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:51.903200 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:51.903174    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 15:54:52.676748 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:52.676709    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" event={"ID":"ac17e449-5699-4e32-8f9c-165ef0cbc298","Type":"ContainerStarted","Data":"33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66"}
Nov 01 15:54:52.677245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:52.676770    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" event={"ID":"ac17e449-5699-4e32-8f9c-165ef0cbc298","Type":"ContainerStarted","Data":"15cdcdcaa3ee019fa846835ecc22c071f814a40da269eab4db5525552b1f3e17"}
Nov 01 15:54:52.846206 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:52.846145    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" podStartSLOduration=1.8461326200000001 podStartE2EDuration="1.84613262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:52.845277746 +0000 UTC m=+5833.010763043" watchObservedRunningTime="2026-11-01 15:54:52.84613262 +0000 UTC m=+5833.011617917"
Nov 01 15:54:53.476367 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:53.476330    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:53.477667 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:53.477643    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 01 15:54:53.477802 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:53.477685    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 01 15:54:54.476771 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:54.476737    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 01 15:54:54.477242 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:54.476793    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 01 15:54:55.475808 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:55.475774    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 01 15:54:55.475989 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:55.475832    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 01 15:54:56.476587 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:56.476546    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 01 15:54:56.477104 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:56.476633    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 01 15:54:57.476149 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:57.476111    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 01 15:54:57.476322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:57.476162    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 01 15:54:58.480528 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:58.480496    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:58.480932 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:58.480851    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:54:58.481509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:54:58.481484    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:55:05.148810 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:55:05.148770    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:55:35.147523 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:55:35.147447    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:56:05.147618 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:05.147582    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:56:08.457841 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.457805    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"]
Nov 01 15:56:08.458332 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.458004    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m" podUID="b7254e30-e914-477c-a827-dc651fbd374d" containerName="maas-api" containerID="cri-o://776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4" gracePeriod=30
Nov 01 15:56:08.740026 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.740002    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:56:08.896885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.896850    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls\") pod \"b7254e30-e914-477c-a827-dc651fbd374d\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") "
Nov 01 15:56:08.897071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.896925    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-metrics-certs\") pod \"b7254e30-e914-477c-a827-dc651fbd374d\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") "
Nov 01 15:56:08.897071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.896963    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7254e30-e914-477c-a827-dc651fbd374d-kube-api-access-sgqqv\" (UniqueName: \"kubernetes.io/projected/b7254e30-e914-477c-a827-dc651fbd374d-kube-api-access-sgqqv\") pod \"b7254e30-e914-477c-a827-dc651fbd374d\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") "
Nov 01 15:56:08.897071 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.897048    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7254e30-e914-477c-a827-dc651fbd374d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7254e30-e914-477c-a827-dc651fbd374d-openshift-service-ca\") pod \"b7254e30-e914-477c-a827-dc651fbd374d\" (UID: \"b7254e30-e914-477c-a827-dc651fbd374d\") "
Nov 01 15:56:08.897467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.897436    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7254e30-e914-477c-a827-dc651fbd374d-openshift-service-ca" pod "b7254e30-e914-477c-a827-dc651fbd374d" (UID: "b7254e30-e914-477c-a827-dc651fbd374d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:08.899036 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.899004    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-metrics-certs" pod "b7254e30-e914-477c-a827-dc651fbd374d" (UID: "b7254e30-e914-477c-a827-dc651fbd374d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:56:08.899173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.899106    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7254e30-e914-477c-a827-dc651fbd374d-kube-api-access-sgqqv" pod "b7254e30-e914-477c-a827-dc651fbd374d" (UID: "b7254e30-e914-477c-a827-dc651fbd374d"). InnerVolumeSpecName "kube-api-access-sgqqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:08.899173 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.899136    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls" pod "b7254e30-e914-477c-a827-dc651fbd374d" (UID: "b7254e30-e914-477c-a827-dc651fbd374d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:56:08.998933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.998840    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:08.998933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.998882    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7254e30-e914-477c-a827-dc651fbd374d-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:08.998933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.998898    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgqqv\" (UniqueName: \"kubernetes.io/projected/b7254e30-e914-477c-a827-dc651fbd374d-kube-api-access-sgqqv\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:08.998933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:08.998913    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7254e30-e914-477c-a827-dc651fbd374d-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:09.077668 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.077628    5423 generic.go:356] "Generic (PLEG): container finished" podID="b7254e30-e914-477c-a827-dc651fbd374d" containerID="776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4" exitCode=0
Nov 01 15:56:09.077846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.077676    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"
Nov 01 15:56:09.077846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.077701    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m" event={"ID":"b7254e30-e914-477c-a827-dc651fbd374d","Type":"ContainerDied","Data":"776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4"}
Nov 01 15:56:09.077846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.077743    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m" event={"ID":"b7254e30-e914-477c-a827-dc651fbd374d","Type":"ContainerDied","Data":"30195e83d9e9738a9f95cbc14bb8547c74ad65efedadb5150ac361a392b259e6"}
Nov 01 15:56:09.077846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.077767    5423 scope.go:122] "RemoveContainer" containerID="776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4"
Nov 01 15:56:09.087462 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.087411    5423 scope.go:122] "RemoveContainer" containerID="776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4"
Nov 01 15:56:09.087692 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:09.087661    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4\": container with ID starting with 776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4 not found: ID does not exist" containerID="776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4"
Nov 01 15:56:09.087799 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.087696    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4"} err="failed to get container status \"776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4\": rpc error: code = NotFound desc = could not find container \"776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4\": container with ID starting with 776c46b9af3e8ea42c13cb27b264ca3b88a5356219cebc8d867d1ac1ea2687a4 not found: ID does not exist"
Nov 01 15:56:09.242820 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.242790    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"]
Nov 01 15:56:09.294066 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.293986    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-dual-3436ba-c4fdd7f5b-j9x8m"]
Nov 01 15:56:09.959969 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:09.959934    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7254e30-e914-477c-a827-dc651fbd374d" path="/var/lib/kubelet/pods/b7254e30-e914-477c-a827-dc651fbd374d/volumes"
Nov 01 15:56:12.872986 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:12.872951    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:56:13.227734 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.227640    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"]
Nov 01 15:56:13.227916 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.227881    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy" containerID="cri-o://255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7" gracePeriod=30
Nov 01 15:56:13.239849 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.239825    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"]
Nov 01 15:56:13.240050 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.240025    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy" containerID="cri-o://33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66" gracePeriod=30
Nov 01 15:56:13.272788 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.272758    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"]
Nov 01 15:56:13.272967 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.272947    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy" containerID="cri-o://b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc" gracePeriod=30
Nov 01 15:56:13.278479 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.278455    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"]
Nov 01 15:56:13.278646 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.278626    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy" containerID="cri-o://c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e" gracePeriod=30
Nov 01 15:56:13.375263 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.375226    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:56:13.375467 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.375293    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:56:13.441583 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.441542    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:56:13.441745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.441601    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:56:13.483509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.483426    5423 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:56:13.483509 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:13.483479    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:56:18.539287 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.539263    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:56:18.592396 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.592361    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:56:18.603951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.603928    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:56:18.607187 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.607166    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:56:18.703825 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.703798    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-kube-api-access-jdd7k\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-kube-api-access-jdd7k\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.703949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.703833    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-certs\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.703949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.703858    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-envoy\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.703949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.703883    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-kube-api-access-mzgg2\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-kube-api-access-mzgg2\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.703949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.703912    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-podinfo\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.703949 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.703934    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-envoy\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.703958    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-kube-api-access-krnm5\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-kube-api-access-krnm5\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.703996    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-socket\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704032    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-data\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704060    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-socket\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704098    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-credential-socket\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704128    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-credential-socket\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704142    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-certs" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704156    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-kube-api-access-4pcjs\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-kube-api-access-4pcjs\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704185    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istiod-ca-cert\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.704232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704219    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac17e449-5699-4e32-8f9c-165ef0cbc298-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac17e449-5699-4e32-8f9c-165ef0cbc298-istiod-ca-cert\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704246    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-envoy\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704301    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istiod-ca-cert\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704333    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-credential-socket\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704362    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-podinfo\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704410    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-socket\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704445    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-token\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-token\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704481    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-socket\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704519    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-certs\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704558    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-envoy\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704599    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-data\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704648    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-token\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704694    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-podinfo\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704722    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-credential-socket\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704767    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-certs\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704802    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-token\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-token\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.704842 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704832    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-token\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-token\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.705812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704871    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istiod-ca-cert\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.705812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704900    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-podinfo\") pod \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\" (UID: \"e5ec076b-ab68-4533-83c5-95f4b980b9fc\") "
Nov 01 15:56:18.705812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704952    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-data\") pod \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\" (UID: \"1328aa87-1471-4bf0-8304-6fbe7e236b7e\") "
Nov 01 15:56:18.705812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.705618    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-data\") pod \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\" (UID: \"b5384fc5-82e9-42b4-a0d6-d07469d9b048\") "
Nov 01 15:56:18.705812 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.705675    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-certs\") pod \"ac17e449-5699-4e32-8f9c-165ef0cbc298\" (UID: \"ac17e449-5699-4e32-8f9c-165ef0cbc298\") "
Nov 01 15:56:18.706614 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.706581    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.707524 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704379    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-data" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.707690 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704594    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-socket" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.707785 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704891    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-socket" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.707868 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.704921    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istiod-ca-cert" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:18.707955 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.705095    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-credential-socket" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.705269    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istiod-ca-cert" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:18.708045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.705293    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac17e449-5699-4e32-8f9c-165ef0cbc298-istiod-ca-cert" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:18.708045 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.705805    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-socket" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.705844    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-credential-socket" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.707038    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-certs" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.707243    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-data" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.707496    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-data" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.708279    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-socket" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708434 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.708369    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-kube-api-access-krnm5" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "kube-api-access-krnm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.708917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.708445    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-envoy" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.708589    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-credential-socket" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.708917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.708799    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-certs" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.709245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.709222    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-credential-socket" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.709607 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.709586    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-certs" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.709939 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.709921    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-data" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.710085 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.709983    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istiod-ca-cert" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:18.711291 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.711245    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-token" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.711766 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.711731    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-podinfo" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:56:18.711978 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.711948    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-kube-api-access-4pcjs" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "kube-api-access-4pcjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.713826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.713373    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-token" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.713826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.713689    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-kube-api-access-jdd7k" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "kube-api-access-jdd7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.713826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.713709    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-podinfo" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:56:18.713826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.713748    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-kube-api-access-mzgg2" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "kube-api-access-mzgg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.713826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.713793    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-envoy" pod "b5384fc5-82e9-42b4-a0d6-d07469d9b048" (UID: "b5384fc5-82e9-42b4-a0d6-d07469d9b048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.713826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.713802    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-token" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.713826 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.713806    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-envoy" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.714265 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.714055    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-podinfo" pod "ac17e449-5699-4e32-8f9c-165ef0cbc298" (UID: "ac17e449-5699-4e32-8f9c-165ef0cbc298"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:56:18.714931 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.714894    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-podinfo" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:56:18.715166 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.715137    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-envoy" pod "1328aa87-1471-4bf0-8304-6fbe7e236b7e" (UID: "1328aa87-1471-4bf0-8304-6fbe7e236b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.715718 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.715685    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-token" pod "e5ec076b-ab68-4533-83c5-95f4b980b9fc" (UID: "e5ec076b-ab68-4533-83c5-95f4b980b9fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.807322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807290    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807317    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807322 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807327    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807336    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807344    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807352    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807361    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807369    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807377    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807402    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807410    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807418    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807426    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807436    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807444    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807453    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807463    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdd7k\" (UniqueName: \"kubernetes.io/projected/e5ec076b-ab68-4533-83c5-95f4b980b9fc-kube-api-access-jdd7k\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807472    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807482    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzgg2\" (UniqueName: \"kubernetes.io/projected/b5384fc5-82e9-42b4-a0d6-d07469d9b048-kube-api-access-mzgg2\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807490    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807498    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807507    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krnm5\" (UniqueName: \"kubernetes.io/projected/ac17e449-5699-4e32-8f9c-165ef0cbc298-kube-api-access-krnm5\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807515    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807525    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807534    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807542    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807551    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5384fc5-82e9-42b4-a0d6-d07469d9b048-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807559    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pcjs\" (UniqueName: \"kubernetes.io/projected/1328aa87-1471-4bf0-8304-6fbe7e236b7e-kube-api-access-4pcjs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807567    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.807565 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807576    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac17e449-5699-4e32-8f9c-165ef0cbc298-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.808428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807584    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ec076b-ab68-4533-83c5-95f4b980b9fc-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.808428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807593    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5384fc5-82e9-42b4-a0d6-d07469d9b048-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.808428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807601    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1328aa87-1471-4bf0-8304-6fbe7e236b7e-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.808428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807610    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac17e449-5699-4e32-8f9c-165ef0cbc298-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.808428 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:18.807617    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac17e449-5699-4e32-8f9c-165ef0cbc298-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:19.129968 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.129890    5423 generic.go:356] "Generic (PLEG): container finished" podID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerID="33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66" exitCode=0
Nov 01 15:56:19.129968 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.129939    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"
Nov 01 15:56:19.130174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.129976    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" event={"ID":"ac17e449-5699-4e32-8f9c-165ef0cbc298","Type":"ContainerDied","Data":"33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66"}
Nov 01 15:56:19.130174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.130029    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd" event={"ID":"ac17e449-5699-4e32-8f9c-165ef0cbc298","Type":"ContainerDied","Data":"15cdcdcaa3ee019fa846835ecc22c071f814a40da269eab4db5525552b1f3e17"}
Nov 01 15:56:19.130174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.130045    5423 scope.go:122] "RemoveContainer" containerID="33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66"
Nov 01 15:56:19.131762 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.131742    5423 generic.go:356] "Generic (PLEG): container finished" podID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerID="255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7" exitCode=0
Nov 01 15:56:19.131884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.131794    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"
Nov 01 15:56:19.131884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.131810    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" event={"ID":"1328aa87-1471-4bf0-8304-6fbe7e236b7e","Type":"ContainerDied","Data":"255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7"}
Nov 01 15:56:19.131884 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.131848    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq" event={"ID":"1328aa87-1471-4bf0-8304-6fbe7e236b7e","Type":"ContainerDied","Data":"f165251565d21f17368ed90647d1a7d871cfe27741e170ec6323e3b61f2e75f8"}
Nov 01 15:56:19.133308 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.133283    5423 generic.go:356] "Generic (PLEG): container finished" podID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerID="b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc" exitCode=0
Nov 01 15:56:19.133431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.133315    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" event={"ID":"e5ec076b-ab68-4533-83c5-95f4b980b9fc","Type":"ContainerDied","Data":"b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc"}
Nov 01 15:56:19.133431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.133339    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"
Nov 01 15:56:19.133431 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.133363    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn" event={"ID":"e5ec076b-ab68-4533-83c5-95f4b980b9fc","Type":"ContainerDied","Data":"7f9a436dee5938e281afe8c6123f0a27c6e275551a30ec900c7f938fc73596d2"}
Nov 01 15:56:19.139739 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.139507    5423 scope.go:122] "RemoveContainer" containerID="33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66"
Nov 01 15:56:19.140102 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:19.140078    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66\": container with ID starting with 33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66 not found: ID does not exist" containerID="33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66"
Nov 01 15:56:19.140176 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.140114    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66"} err="failed to get container status \"33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66\": rpc error: code = NotFound desc = could not find container \"33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66\": container with ID starting with 33de20948a150fd4692a2208284d93adac682b9c0b59bf63b228c6709563dd66 not found: ID does not exist"
Nov 01 15:56:19.140176 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.140140    5423 scope.go:122] "RemoveContainer" containerID="255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7"
Nov 01 15:56:19.140313 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.140292    5423 generic.go:356] "Generic (PLEG): container finished" podID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerID="c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e" exitCode=0
Nov 01 15:56:19.140373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.140323    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" event={"ID":"b5384fc5-82e9-42b4-a0d6-d07469d9b048","Type":"ContainerDied","Data":"c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e"}
Nov 01 15:56:19.140373 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.140356    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5" event={"ID":"b5384fc5-82e9-42b4-a0d6-d07469d9b048","Type":"ContainerDied","Data":"7416c5c6257bff941688ddb0a960d4fe231f548c0c84a9f6b5a85070b4c8ee5e"}
Nov 01 15:56:19.140484 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.140460    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"
Nov 01 15:56:19.150238 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.150222    5423 scope.go:122] "RemoveContainer" containerID="3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe"
Nov 01 15:56:19.157952 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.157936    5423 scope.go:122] "RemoveContainer" containerID="255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7"
Nov 01 15:56:19.158197 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:19.158179    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7\": container with ID starting with 255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7 not found: ID does not exist" containerID="255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7"
Nov 01 15:56:19.158245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.158206    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7"} err="failed to get container status \"255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7\": rpc error: code = NotFound desc = could not find container \"255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7\": container with ID starting with 255a243f2692d5878d327aeffd03d11723cd9b35a282d76a0331ad3550247db7 not found: ID does not exist"
Nov 01 15:56:19.158245 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.158222    5423 scope.go:122] "RemoveContainer" containerID="3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe"
Nov 01 15:56:19.158433 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:19.158416    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe\": container with ID starting with 3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe not found: ID does not exist" containerID="3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe"
Nov 01 15:56:19.158482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.158438    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe"} err="failed to get container status \"3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe\": rpc error: code = NotFound desc = could not find container \"3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe\": container with ID starting with 3abc9f4d022fda1e06ede5361e88ac7c7e5d1c638ed98b0e1047e588abc8f2fe not found: ID does not exist"
Nov 01 15:56:19.158482 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.158451    5423 scope.go:122] "RemoveContainer" containerID="b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc"
Nov 01 15:56:19.165852 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.165827    5423 scope.go:122] "RemoveContainer" containerID="b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc"
Nov 01 15:56:19.166105 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:19.166089    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc\": container with ID starting with b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc not found: ID does not exist" containerID="b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc"
Nov 01 15:56:19.166157 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.166112    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc"} err="failed to get container status \"b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc\": rpc error: code = NotFound desc = could not find container \"b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc\": container with ID starting with b610a757892b669563e1634810201bef08a67bb443f4c193d03a4b19cad637bc not found: ID does not exist"
Nov 01 15:56:19.166157 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.166135    5423 scope.go:122] "RemoveContainer" containerID="c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e"
Nov 01 15:56:19.173255 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.173237    5423 scope.go:122] "RemoveContainer" containerID="c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e"
Nov 01 15:56:19.173519 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:19.173501    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e\": container with ID starting with c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e not found: ID does not exist" containerID="c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e"
Nov 01 15:56:19.173566 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.173523    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e"} err="failed to get container status \"c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e\": rpc error: code = NotFound desc = could not find container \"c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e\": container with ID starting with c7dd982272780e6d9cae149485a56b0af07d35286298e51a2c5948e8cc557c0e not found: ID does not exist"
Nov 01 15:56:19.297571 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.297540    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"]
Nov 01 15:56:19.300584 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.300563    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-hjksq"]
Nov 01 15:56:19.411347 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.411219    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"]
Nov 01 15:56:19.414418 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.414377    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-qgps5"]
Nov 01 15:56:19.526331 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.526298    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"]
Nov 01 15:56:19.529257 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.529234    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-4wsnd"]
Nov 01 15:56:19.640981 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.640951    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"]
Nov 01 15:56:19.644112 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.644086    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-9rdfn"]
Nov 01 15:56:19.959427 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.959366    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" path="/var/lib/kubelet/pods/1328aa87-1471-4bf0-8304-6fbe7e236b7e/volumes"
Nov 01 15:56:19.959914 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.959900    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" path="/var/lib/kubelet/pods/ac17e449-5699-4e32-8f9c-165ef0cbc298/volumes"
Nov 01 15:56:19.960311 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.960298    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" path="/var/lib/kubelet/pods/b5384fc5-82e9-42b4-a0d6-d07469d9b048/volumes"
Nov 01 15:56:19.960988 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:19.960973    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" path="/var/lib/kubelet/pods/e5ec076b-ab68-4533-83c5-95f4b980b9fc/volumes"
Nov 01 15:56:31.014020 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:31.013988    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"]
Nov 01 15:56:32.961525 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.961495    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-57b57bf698-jjhrw"]
Nov 01 15:56:32.961986 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.961968    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy"
Nov 01 15:56:32.961986 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.961987    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy"
Nov 01 15:56:32.962087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962018    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy"
Nov 01 15:56:32.962087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962027    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy"
Nov 01 15:56:32.962087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962046    5423 cpu_manager.go:414] "containerMap: removing container" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy"
Nov 01 15:56:32.962087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962054    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy"
Nov 01 15:56:32.962087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962066    5423 cpu_manager.go:414] "containerMap: removing container" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy"
Nov 01 15:56:32.962087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962073    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy"
Nov 01 15:56:32.962087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962085    5423 cpu_manager.go:414] "containerMap: removing container" podUID="b7254e30-e914-477c-a827-dc651fbd374d" containerName="maas-api"
Nov 01 15:56:32.962087 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962089    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7254e30-e914-477c-a827-dc651fbd374d" containerName="maas-api"
Nov 01 15:56:32.962417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962166    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy"
Nov 01 15:56:32.962417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962180    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5384fc5-82e9-42b4-a0d6-d07469d9b048" containerName="istio-proxy"
Nov 01 15:56:32.962417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962190    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy"
Nov 01 15:56:32.962417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962198    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ec076b-ab68-4533-83c5-95f4b980b9fc" containerName="istio-proxy"
Nov 01 15:56:32.962417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962203    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac17e449-5699-4e32-8f9c-165ef0cbc298" containerName="istio-proxy"
Nov 01 15:56:32.962417 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.962210    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7254e30-e914-477c-a827-dc651fbd374d" containerName="maas-api"
Nov 01 15:56:32.965523 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:32.965504    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:33.021839 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.021798    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-57b57bf698-jjhrw"]
Nov 01 15:56:33.038337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.038299    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbl5\" (UniqueName: \"kubernetes.io/projected/1120ae17-ca00-496f-9216-3db30096e3b8-kube-api-access-wxbl5\") pod \"payload-processing-57b57bf698-jjhrw\" (UID: \"1120ae17-ca00-496f-9216-3db30096e3b8\") " pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:33.038600 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.038575    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120ae17-ca00-496f-9216-3db30096e3b8-plugins-config-volume\") pod \"payload-processing-57b57bf698-jjhrw\" (UID: \"1120ae17-ca00-496f-9216-3db30096e3b8\") " pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:33.140065 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.140033    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120ae17-ca00-496f-9216-3db30096e3b8-plugins-config-volume\") pod \"payload-processing-57b57bf698-jjhrw\" (UID: \"1120ae17-ca00-496f-9216-3db30096e3b8\") " pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:33.140232 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.140080    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbl5\" (UniqueName: \"kubernetes.io/projected/1120ae17-ca00-496f-9216-3db30096e3b8-kube-api-access-wxbl5\") pod \"payload-processing-57b57bf698-jjhrw\" (UID: \"1120ae17-ca00-496f-9216-3db30096e3b8\") " pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:33.140832 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.140811    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120ae17-ca00-496f-9216-3db30096e3b8-plugins-config-volume\") pod \"payload-processing-57b57bf698-jjhrw\" (UID: \"1120ae17-ca00-496f-9216-3db30096e3b8\") " pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:33.200182 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.200149    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbl5\" (UniqueName: \"kubernetes.io/projected/1120ae17-ca00-496f-9216-3db30096e3b8-kube-api-access-wxbl5\") pod \"payload-processing-57b57bf698-jjhrw\" (UID: \"1120ae17-ca00-496f-9216-3db30096e3b8\") " pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:33.278101 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.278028    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:33.646451 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:33.646428    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-57b57bf698-jjhrw"]
Nov 01 15:56:33.648818 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:56:33.648789    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1120ae17_ca00_496f_9216_3db30096e3b8.slice/crio-88581f73da6ce78ef4664d87ce706a0b4c43ab90007c317582de4a50593aba9d WatchSource:0}: Error finding container 88581f73da6ce78ef4664d87ce706a0b4c43ab90007c317582de4a50593aba9d: Status 404 returned error can't find the container with id 88581f73da6ce78ef4664d87ce706a0b4c43ab90007c317582de4a50593aba9d
Nov 01 15:56:34.219871 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:34.219835    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw" event={"ID":"1120ae17-ca00-496f-9216-3db30096e3b8","Type":"ContainerStarted","Data":"e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b"}
Nov 01 15:56:34.220316 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:34.219888    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw" event={"ID":"1120ae17-ca00-496f-9216-3db30096e3b8","Type":"ContainerStarted","Data":"88581f73da6ce78ef4664d87ce706a0b4c43ab90007c317582de4a50593aba9d"}
Nov 01 15:56:34.220316 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:34.219939    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:34.336666 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:34.336615    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw" podStartSLOduration=2.336601162 podStartE2EDuration="2.336601162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:34.334853457 +0000 UTC m=+5934.500338754" watchObservedRunningTime="2026-11-01 15:56:34.336601162 +0000 UTC m=+5934.502086454"
Nov 01 15:56:35.147735 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:35.147696    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:56:36.014951 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.014898    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf" podUID="f6ac5457-19e0-476a-9137-052ded3d5a13" containerName="payload-pre-processing" containerID="cri-o://7f84b362d7f1ec1e29148959a8c7cd508726db0b23d4c8b47390145fc8155b4e" gracePeriod=25
Nov 01 15:56:36.230458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.230429    5423 generic.go:356] "Generic (PLEG): container finished" podID="f6ac5457-19e0-476a-9137-052ded3d5a13" containerID="7f84b362d7f1ec1e29148959a8c7cd508726db0b23d4c8b47390145fc8155b4e" exitCode=0
Nov 01 15:56:36.230590 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.230499    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf" event={"ID":"f6ac5457-19e0-476a-9137-052ded3d5a13","Type":"ContainerDied","Data":"7f84b362d7f1ec1e29148959a8c7cd508726db0b23d4c8b47390145fc8155b4e"}
Nov 01 15:56:36.260655 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.260634    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 15:56:36.369040 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.369013    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ac5457-19e0-476a-9137-052ded3d5a13-kube-api-access-tmmfj\" (UniqueName: \"kubernetes.io/projected/f6ac5457-19e0-476a-9137-052ded3d5a13-kube-api-access-tmmfj\") pod \"f6ac5457-19e0-476a-9137-052ded3d5a13\" (UID: \"f6ac5457-19e0-476a-9137-052ded3d5a13\") "
Nov 01 15:56:36.369186 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.369104    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ac5457-19e0-476a-9137-052ded3d5a13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ac5457-19e0-476a-9137-052ded3d5a13-plugins-config-volume\") pod \"f6ac5457-19e0-476a-9137-052ded3d5a13\" (UID: \"f6ac5457-19e0-476a-9137-052ded3d5a13\") "
Nov 01 15:56:36.369458 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.369437    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ac5457-19e0-476a-9137-052ded3d5a13-plugins-config-volume" pod "f6ac5457-19e0-476a-9137-052ded3d5a13" (UID: "f6ac5457-19e0-476a-9137-052ded3d5a13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:36.371134 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.371110    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ac5457-19e0-476a-9137-052ded3d5a13-kube-api-access-tmmfj" pod "f6ac5457-19e0-476a-9137-052ded3d5a13" (UID: "f6ac5457-19e0-476a-9137-052ded3d5a13"). InnerVolumeSpecName "kube-api-access-tmmfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:36.470274 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.470245    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ac5457-19e0-476a-9137-052ded3d5a13-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:36.470274 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:36.470270    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmmfj\" (UniqueName: \"kubernetes.io/projected/f6ac5457-19e0-476a-9137-052ded3d5a13-kube-api-access-tmmfj\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:37.235863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:37.235838    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"
Nov 01 15:56:37.235863 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:37.235850    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf" event={"ID":"f6ac5457-19e0-476a-9137-052ded3d5a13","Type":"ContainerDied","Data":"02ff67e51e4372441355232683eeefd67f456da84b74ee34b16006ac410dd819"}
Nov 01 15:56:37.236412 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:37.235910    5423 scope.go:122] "RemoveContainer" containerID="7f84b362d7f1ec1e29148959a8c7cd508726db0b23d4c8b47390145fc8155b4e"
Nov 01 15:56:37.401625 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:37.401589    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"]
Nov 01 15:56:37.452633 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:37.452603    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vvxtf"]
Nov 01 15:56:37.962750 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:37.962715    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ac5457-19e0-476a-9137-052ded3d5a13" path="/var/lib/kubelet/pods/f6ac5457-19e0-476a-9137-052ded3d5a13/volumes"
Nov 01 15:56:45.227602 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:45.227572    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:56:45.395357 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:45.395328    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"]
Nov 01 15:56:47.425349 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.425308    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-dd858cb66-68dsg"]
Nov 01 15:56:47.425744 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.425729    5423 cpu_manager.go:414] "containerMap: removing container" podUID="f6ac5457-19e0-476a-9137-052ded3d5a13" containerName="payload-pre-processing"
Nov 01 15:56:47.425784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.425745    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ac5457-19e0-476a-9137-052ded3d5a13" containerName="payload-pre-processing"
Nov 01 15:56:47.425784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.425770    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy"
Nov 01 15:56:47.425784 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.425776    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1328aa87-1471-4bf0-8304-6fbe7e236b7e" containerName="istio-proxy"
Nov 01 15:56:47.425879 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.425829    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ac5457-19e0-476a-9137-052ded3d5a13" containerName="payload-pre-processing"
Nov 01 15:56:47.430181 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.430166    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:47.469815 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.469786    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2678e74b-8388-4f25-b832-934ccd98d338-plugins-config-volume\") pod \"payload-processing-dd858cb66-68dsg\" (UID: \"2678e74b-8388-4f25-b832-934ccd98d338\") " pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:47.469948 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.469851    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9db4\" (UniqueName: \"kubernetes.io/projected/2678e74b-8388-4f25-b832-934ccd98d338-kube-api-access-t9db4\") pod \"payload-processing-dd858cb66-68dsg\" (UID: \"2678e74b-8388-4f25-b832-934ccd98d338\") " pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:47.486085 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.486056    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-dd858cb66-68dsg"]
Nov 01 15:56:47.570708 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.570680    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2678e74b-8388-4f25-b832-934ccd98d338-plugins-config-volume\") pod \"payload-processing-dd858cb66-68dsg\" (UID: \"2678e74b-8388-4f25-b832-934ccd98d338\") " pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:47.570833 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.570751    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9db4\" (UniqueName: \"kubernetes.io/projected/2678e74b-8388-4f25-b832-934ccd98d338-kube-api-access-t9db4\") pod \"payload-processing-dd858cb66-68dsg\" (UID: \"2678e74b-8388-4f25-b832-934ccd98d338\") " pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:47.571444 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.571425    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2678e74b-8388-4f25-b832-934ccd98d338-plugins-config-volume\") pod \"payload-processing-dd858cb66-68dsg\" (UID: \"2678e74b-8388-4f25-b832-934ccd98d338\") " pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:47.589024 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.588994    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-688d45fc-hm5jh"]
Nov 01 15:56:47.592687 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.592671    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:47.628537 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.628514    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9db4\" (UniqueName: \"kubernetes.io/projected/2678e74b-8388-4f25-b832-934ccd98d338-kube-api-access-t9db4\") pod \"payload-processing-dd858cb66-68dsg\" (UID: \"2678e74b-8388-4f25-b832-934ccd98d338\") " pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:47.648885 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.648861    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-688d45fc-hm5jh"]
Nov 01 15:56:47.671506 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.671473    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjlt\" (UniqueName: \"kubernetes.io/projected/ae261db3-a26e-4f0b-ad85-6a487a3faeda-kube-api-access-xmjlt\") pod \"payload-pre-processing-688d45fc-hm5jh\" (UID: \"ae261db3-a26e-4f0b-ad85-6a487a3faeda\") " pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:47.671635 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.671537    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae261db3-a26e-4f0b-ad85-6a487a3faeda-plugins-config-volume\") pod \"payload-pre-processing-688d45fc-hm5jh\" (UID: \"ae261db3-a26e-4f0b-ad85-6a487a3faeda\") " pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:47.743174 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.743104    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:47.772364 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.772338    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjlt\" (UniqueName: \"kubernetes.io/projected/ae261db3-a26e-4f0b-ad85-6a487a3faeda-kube-api-access-xmjlt\") pod \"payload-pre-processing-688d45fc-hm5jh\" (UID: \"ae261db3-a26e-4f0b-ad85-6a487a3faeda\") " pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:47.772491 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.772381    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae261db3-a26e-4f0b-ad85-6a487a3faeda-plugins-config-volume\") pod \"payload-pre-processing-688d45fc-hm5jh\" (UID: \"ae261db3-a26e-4f0b-ad85-6a487a3faeda\") " pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:47.772990 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.772972    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae261db3-a26e-4f0b-ad85-6a487a3faeda-plugins-config-volume\") pod \"payload-pre-processing-688d45fc-hm5jh\" (UID: \"ae261db3-a26e-4f0b-ad85-6a487a3faeda\") " pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:47.830159 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.830127    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjlt\" (UniqueName: \"kubernetes.io/projected/ae261db3-a26e-4f0b-ad85-6a487a3faeda-kube-api-access-xmjlt\") pod \"payload-pre-processing-688d45fc-hm5jh\" (UID: \"ae261db3-a26e-4f0b-ad85-6a487a3faeda\") " pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:47.904872 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:47.904848    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:48.112080 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:48.112058    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-dd858cb66-68dsg"]
Nov 01 15:56:48.114263 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:56:48.114234    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2678e74b_8388_4f25_b832_934ccd98d338.slice/crio-013897fb81e4fe67328d4076e1ddd2d7a93905c8680d9a65b699f3787db8d401 WatchSource:0}: Error finding container 013897fb81e4fe67328d4076e1ddd2d7a93905c8680d9a65b699f3787db8d401: Status 404 returned error can't find the container with id 013897fb81e4fe67328d4076e1ddd2d7a93905c8680d9a65b699f3787db8d401
Nov 01 15:56:48.276860 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:48.276830    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-688d45fc-hm5jh"]
Nov 01 15:56:48.279894 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:56:48.279870    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae261db3_a26e_4f0b_ad85_6a487a3faeda.slice/crio-7a220a7ee46382f58ecb7ec1c7ebde735abf2beead8f24dd32d27361e0c6682b WatchSource:0}: Error finding container 7a220a7ee46382f58ecb7ec1c7ebde735abf2beead8f24dd32d27361e0c6682b: Status 404 returned error can't find the container with id 7a220a7ee46382f58ecb7ec1c7ebde735abf2beead8f24dd32d27361e0c6682b
Nov 01 15:56:48.287882 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:48.287854    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-dd858cb66-68dsg" event={"ID":"2678e74b-8388-4f25-b832-934ccd98d338","Type":"ContainerStarted","Data":"bc8f68a2fd3bb67b3113a04d2daba7b7fc19038b3011a16079569816924b8216"}
Nov 01 15:56:48.287993 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:48.287913    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-dd858cb66-68dsg" event={"ID":"2678e74b-8388-4f25-b832-934ccd98d338","Type":"ContainerStarted","Data":"013897fb81e4fe67328d4076e1ddd2d7a93905c8680d9a65b699f3787db8d401"}
Nov 01 15:56:48.287993 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:48.287969    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:48.289046 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:48.289024    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh" event={"ID":"ae261db3-a26e-4f0b-ad85-6a487a3faeda","Type":"ContainerStarted","Data":"7a220a7ee46382f58ecb7ec1c7ebde735abf2beead8f24dd32d27361e0c6682b"}
Nov 01 15:56:48.447976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:48.447887    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-dd858cb66-68dsg" podStartSLOduration=1.447872728 podStartE2EDuration="1.447872728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:48.44686935 +0000 UTC m=+5948.612354650" watchObservedRunningTime="2026-11-01 15:56:48.447872728 +0000 UTC m=+5948.613358020"
Nov 01 15:56:49.295718 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:49.295678    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh" event={"ID":"ae261db3-a26e-4f0b-ad85-6a487a3faeda","Type":"ContainerStarted","Data":"debdf414b76b4331c89def30d64cb049c6005ea215695042a2fbedf4ba1a2753"}
Nov 01 15:56:49.412994 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:49.412900    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh" podStartSLOduration=2.412887891 podStartE2EDuration="2.412887891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:49.411553092 +0000 UTC m=+5949.577038389" watchObservedRunningTime="2026-11-01 15:56:49.412887891 +0000 UTC m=+5949.578373219"
Nov 01 15:56:50.299831 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.299801    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:56:50.395976 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.395941    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5" podUID="0162fda0-7a65-4c47-8aa7-46dd184f8cb5" containerName="payload-processing" containerID="cri-o://94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04" gracePeriod=25
Nov 01 15:56:50.644704 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.644679    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 15:56:50.698278 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.698250    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-kube-api-access-m88v7\" (UniqueName: \"kubernetes.io/projected/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-kube-api-access-m88v7\") pod \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\" (UID: \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\") "
Nov 01 15:56:50.698436 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.698420    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-plugins-config-volume\") pod \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\" (UID: \"0162fda0-7a65-4c47-8aa7-46dd184f8cb5\") "
Nov 01 15:56:50.698745 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.698725    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-plugins-config-volume" pod "0162fda0-7a65-4c47-8aa7-46dd184f8cb5" (UID: "0162fda0-7a65-4c47-8aa7-46dd184f8cb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:50.700224 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.700204    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-kube-api-access-m88v7" pod "0162fda0-7a65-4c47-8aa7-46dd184f8cb5" (UID: "0162fda0-7a65-4c47-8aa7-46dd184f8cb5"). InnerVolumeSpecName "kube-api-access-m88v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:50.799795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.799764    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:50.799795 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:50.799790    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m88v7\" (UniqueName: \"kubernetes.io/projected/0162fda0-7a65-4c47-8aa7-46dd184f8cb5-kube-api-access-m88v7\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:51.305459 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.305418    5423 generic.go:356] "Generic (PLEG): container finished" podID="0162fda0-7a65-4c47-8aa7-46dd184f8cb5" containerID="94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04" exitCode=0
Nov 01 15:56:51.305880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.305516    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"
Nov 01 15:56:51.305880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.305548    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5" event={"ID":"0162fda0-7a65-4c47-8aa7-46dd184f8cb5","Type":"ContainerDied","Data":"94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04"}
Nov 01 15:56:51.305880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.305594    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-wc2s5" event={"ID":"0162fda0-7a65-4c47-8aa7-46dd184f8cb5","Type":"ContainerDied","Data":"7a5b51c1c467c55b0ee4dc2b29abbb40eda2b6e7cde014c49009884b163155d5"}
Nov 01 15:56:51.305880 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.305614    5423 scope.go:122] "RemoveContainer" containerID="94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04"
Nov 01 15:56:51.314613 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.314594    5423 scope.go:122] "RemoveContainer" containerID="94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04"
Nov 01 15:56:51.314864 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:56:51.314846    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04\": container with ID starting with 94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04 not found: ID does not exist" containerID="94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04"
Nov 01 15:56:51.314911 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.314872    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04"} err="failed to get container status \"94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04\": rpc error: code = NotFound desc = could not find container \"94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04\": container with ID starting with 94cb30619d557953ea8bd740a945b2d1dbf86f0563af9cf6cefa52ace7e09e04 not found: ID does not exist"
Nov 01 15:56:51.472017 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.471988    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"]
Nov 01 15:56:51.522931 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.522900    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wc2s5"]
Nov 01 15:56:51.958528 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:51.958489    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0162fda0-7a65-4c47-8aa7-46dd184f8cb5" path="/var/lib/kubelet/pods/0162fda0-7a65-4c47-8aa7-46dd184f8cb5/volumes"
Nov 01 15:56:59.298716 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:59.298684    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-dd858cb66-68dsg"
Nov 01 15:56:59.466581 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:56:59.466547    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-57b57bf698-jjhrw"]
Nov 01 15:57:01.308586 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:01.308561    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-688d45fc-hm5jh"
Nov 01 15:57:04.467813 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:04.467769    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw" podUID="1120ae17-ca00-496f-9216-3db30096e3b8" containerName="payload-processing" containerID="cri-o://e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b" gracePeriod=25
Nov 01 15:57:04.719362 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:04.719309    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:57:04.829736 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:04.829698    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1120ae17-ca00-496f-9216-3db30096e3b8-kube-api-access-wxbl5\" (UniqueName: \"kubernetes.io/projected/1120ae17-ca00-496f-9216-3db30096e3b8-kube-api-access-wxbl5\") pod \"1120ae17-ca00-496f-9216-3db30096e3b8\" (UID: \"1120ae17-ca00-496f-9216-3db30096e3b8\") "
Nov 01 15:57:04.829900 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:04.829787    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1120ae17-ca00-496f-9216-3db30096e3b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120ae17-ca00-496f-9216-3db30096e3b8-plugins-config-volume\") pod \"1120ae17-ca00-496f-9216-3db30096e3b8\" (UID: \"1120ae17-ca00-496f-9216-3db30096e3b8\") "
Nov 01 15:57:04.830219 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:04.830193    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1120ae17-ca00-496f-9216-3db30096e3b8-plugins-config-volume" pod "1120ae17-ca00-496f-9216-3db30096e3b8" (UID: "1120ae17-ca00-496f-9216-3db30096e3b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:57:04.831768 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:04.831737    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1120ae17-ca00-496f-9216-3db30096e3b8-kube-api-access-wxbl5" pod "1120ae17-ca00-496f-9216-3db30096e3b8" (UID: "1120ae17-ca00-496f-9216-3db30096e3b8"). InnerVolumeSpecName "kube-api-access-wxbl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:57:04.930729 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:04.930698    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbl5\" (UniqueName: \"kubernetes.io/projected/1120ae17-ca00-496f-9216-3db30096e3b8-kube-api-access-wxbl5\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:57:04.930729 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:04.930730    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1120ae17-ca00-496f-9216-3db30096e3b8-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:57:05.148018 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:57:05.147985    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:57:05.370334 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.370302    5423 generic.go:356] "Generic (PLEG): container finished" podID="1120ae17-ca00-496f-9216-3db30096e3b8" containerID="e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b" exitCode=0
Nov 01 15:57:05.370517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.370342    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw"
Nov 01 15:57:05.370517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.370379    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw" event={"ID":"1120ae17-ca00-496f-9216-3db30096e3b8","Type":"ContainerDied","Data":"e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b"}
Nov 01 15:57:05.370517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.370446    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-57b57bf698-jjhrw" event={"ID":"1120ae17-ca00-496f-9216-3db30096e3b8","Type":"ContainerDied","Data":"88581f73da6ce78ef4664d87ce706a0b4c43ab90007c317582de4a50593aba9d"}
Nov 01 15:57:05.370517 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.370468    5423 scope.go:122] "RemoveContainer" containerID="e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b"
Nov 01 15:57:05.379698 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.379682    5423 scope.go:122] "RemoveContainer" containerID="e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b"
Nov 01 15:57:05.379953 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:57:05.379932    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b\": container with ID starting with e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b not found: ID does not exist" containerID="e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b"
Nov 01 15:57:05.380008 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.379966    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b"} err="failed to get container status \"e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b\": rpc error: code = NotFound desc = could not find container \"e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b\": container with ID starting with e4aa2ed077242650880229cd1048d30c780ba5cf0de82f2b4c6cd9392034ae3b not found: ID does not exist"
Nov 01 15:57:05.535342 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.535278    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-57b57bf698-jjhrw"]
Nov 01 15:57:05.586192 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.586171    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-57b57bf698-jjhrw"]
Nov 01 15:57:05.958933 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:05.958900    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1120ae17-ca00-496f-9216-3db30096e3b8" path="/var/lib/kubelet/pods/1120ae17-ca00-496f-9216-3db30096e3b8/volumes"
Nov 01 15:57:07.177175 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.177139    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"]
Nov 01 15:57:07.177679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.177621    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1120ae17-ca00-496f-9216-3db30096e3b8" containerName="payload-processing"
Nov 01 15:57:07.177679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.177637    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1120ae17-ca00-496f-9216-3db30096e3b8" containerName="payload-processing"
Nov 01 15:57:07.177679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.177653    5423 cpu_manager.go:414] "containerMap: removing container" podUID="0162fda0-7a65-4c47-8aa7-46dd184f8cb5" containerName="payload-processing"
Nov 01 15:57:07.177679 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.177659    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0162fda0-7a65-4c47-8aa7-46dd184f8cb5" containerName="payload-processing"
Nov 01 15:57:07.177819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.177731    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1120ae17-ca00-496f-9216-3db30096e3b8" containerName="payload-processing"
Nov 01 15:57:07.177819 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.177743    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="0162fda0-7a65-4c47-8aa7-46dd184f8cb5" containerName="payload-processing"
Nov 01 15:57:07.181993 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.181970    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.235636 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.235589    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 01 15:57:07.237215 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.237191    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"]
Nov 01 15:57:07.286700 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.286673    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 15:57:07.286846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.286798    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 15:57:07.286846 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.286838    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4rffg\""
Nov 01 15:57:07.286954 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.286889    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 15:57:07.287005 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.286969    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 15:57:07.352927 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.352891    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.353102 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.352939    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knb6w\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-kube-api-access-knb6w\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.353102 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.353064    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.353102 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.353098    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.454235 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.454155    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.454235 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.454221    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.454533 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.454255    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knb6w\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-kube-api-access-knb6w\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.454533 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.454319    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.454998 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.454967    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.455226 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.455204    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.456761 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.456742    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.512185 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.512149    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knb6w\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-kube-api-access-knb6w\") pod \"payload-processing-external-model-5bcd7997-tpzzf\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:07.794910 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:07.794824    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:08.163452 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:08.163339    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"]
Nov 01 15:57:08.166855 ip-10-0-135-142 kubenswrapper[5423]: W1101 15:57:08.166826    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1314115e_f7e4_4c2e_bf24_1d5602d3fedb.slice/crio-6614b7886705a6cbe94f44e12e7b38487e359a680c3a79f6951e706e2a63b2e0 WatchSource:0}: Error finding container 6614b7886705a6cbe94f44e12e7b38487e359a680c3a79f6951e706e2a63b2e0: Status 404 returned error can't find the container with id 6614b7886705a6cbe94f44e12e7b38487e359a680c3a79f6951e706e2a63b2e0
Nov 01 15:57:08.356448 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:08.356418    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:57:08.386897 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:08.386865    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf" event={"ID":"1314115e-f7e4-4c2e-bf24-1d5602d3fedb","Type":"ContainerStarted","Data":"868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c"}
Nov 01 15:57:08.387077 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:08.386920    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf" event={"ID":"1314115e-f7e4-4c2e-bf24-1d5602d3fedb","Type":"ContainerStarted","Data":"6614b7886705a6cbe94f44e12e7b38487e359a680c3a79f6951e706e2a63b2e0"}
Nov 01 15:57:08.387077 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:08.387027    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:08.503818 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:08.503730    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf" podStartSLOduration=1.503715853 podStartE2EDuration="1.503715853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:08.502439284 +0000 UTC m=+5968.667924585" watchObservedRunningTime="2026-11-01 15:57:08.503715853 +0000 UTC m=+5968.669201187"
Nov 01 15:57:19.396822 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:19.396794    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:57:35.147986 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:57:35.147940    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:57:43.276501 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:43.276425    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:57:43.290088 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:57:43.290063    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 15:58:05.147969 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:58:05.147928    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:58:35.147413 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:58:35.147325    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:58:49.295161 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:49.295124    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"]
Nov 01 15:58:52.070425 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:52.070378    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:58:54.296010 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.295949    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf" podUID="1314115e-f7e4-4c2e-bf24-1d5602d3fedb" containerName="payload-processing" containerID="cri-o://868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c" gracePeriod=25
Nov 01 15:58:54.546238 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.546183    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:58:54.573780 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.573758    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-provider-credentials\") pod \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") "
Nov 01 15:58:54.573950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.573861    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-routing-overlay-volume\") pod \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") "
Nov 01 15:58:54.573950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.573893    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-plugins-config-volume\") pod \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") "
Nov 01 15:58:54.573950 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.573920    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-kube-api-access-knb6w\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-kube-api-access-knb6w\") pod \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\" (UID: \"1314115e-f7e4-4c2e-bf24-1d5602d3fedb\") "
Nov 01 15:58:54.574259 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.574232    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-routing-overlay-volume" pod "1314115e-f7e4-4c2e-bf24-1d5602d3fedb" (UID: "1314115e-f7e4-4c2e-bf24-1d5602d3fedb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:58:54.574337 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.574255    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-plugins-config-volume" pod "1314115e-f7e4-4c2e-bf24-1d5602d3fedb" (UID: "1314115e-f7e4-4c2e-bf24-1d5602d3fedb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:58:54.575917 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.575894    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-provider-credentials" pod "1314115e-f7e4-4c2e-bf24-1d5602d3fedb" (UID: "1314115e-f7e4-4c2e-bf24-1d5602d3fedb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:58:54.576030 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.576007    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-kube-api-access-knb6w" pod "1314115e-f7e4-4c2e-bf24-1d5602d3fedb" (UID: "1314115e-f7e4-4c2e-bf24-1d5602d3fedb"). InnerVolumeSpecName "kube-api-access-knb6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:58:54.675492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.675463    5423 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-routing-overlay-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:58:54.675492 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.675492    5423 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-plugins-config-volume\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:58:54.675669 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.675502    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knb6w\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-kube-api-access-knb6w\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:58:54.675669 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.675511    5423 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1314115e-f7e4-4c2e-bf24-1d5602d3fedb-provider-credentials\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:58:54.883844 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.883808    5423 generic.go:356] "Generic (PLEG): container finished" podID="1314115e-f7e4-4c2e-bf24-1d5602d3fedb" containerID="868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c" exitCode=0
Nov 01 15:58:54.884037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.883855    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"
Nov 01 15:58:54.884037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.883889    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf" event={"ID":"1314115e-f7e4-4c2e-bf24-1d5602d3fedb","Type":"ContainerDied","Data":"868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c"}
Nov 01 15:58:54.884037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.883943    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf" event={"ID":"1314115e-f7e4-4c2e-bf24-1d5602d3fedb","Type":"ContainerDied","Data":"6614b7886705a6cbe94f44e12e7b38487e359a680c3a79f6951e706e2a63b2e0"}
Nov 01 15:58:54.884037 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.883964    5423 scope.go:122] "RemoveContainer" containerID="868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c"
Nov 01 15:58:54.898968 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.898947    5423 scope.go:122] "RemoveContainer" containerID="868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c"
Nov 01 15:58:54.899266 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:58:54.899246    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c\": container with ID starting with 868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c not found: ID does not exist" containerID="868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c"
Nov 01 15:58:54.899317 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:54.899274    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c"} err="failed to get container status \"868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c\": rpc error: code = NotFound desc = could not find container \"868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c\": container with ID starting with 868d50ca5e332ecd88ae34585a382bf4eb2c7c3365d627acd545325cfef8e17c not found: ID does not exist"
Nov 01 15:58:55.050891 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:55.050863    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"]
Nov 01 15:58:55.102221 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:55.102188    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tpzzf"]
Nov 01 15:58:55.958817 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:58:55.958772    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1314115e-f7e4-4c2e-bf24-1d5602d3fedb" path="/var/lib/kubelet/pods/1314115e-f7e4-4c2e-bf24-1d5602d3fedb/volumes"
Nov 01 15:59:01.857413 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:59:01.857368    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:59:05.149794 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:59:05.149757    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:59:19.948808 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:59:19.948775    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:59:31.649056 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:59:31.649024    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 15:59:35.148944 ip-10-0-135-142 kubenswrapper[5423]: E1101 15:59:35.148904    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 15:59:48.971556 ip-10-0-135-142 kubenswrapper[5423]: I1101 15:59:48.971518    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:00:02.072482 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:00:02.072441    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:00:05.148528 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:00:05.148484    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:00:32.435329 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:00:32.435290    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:00:35.189611 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:00:35.189581    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:00:43.069664 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:00:43.069629    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:00:54.664043 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:00:54.663998    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:01:05.147704 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:01:05.147666    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:01:17.746534 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:17.746493    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:01:24.922594 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:24.922490    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gkt8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:01:24.923218 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:24.923181    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podUID="35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:01:26.773326 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:26.773293    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:01:34.757860 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:34.757826    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:01:35.151260 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:01:35.151216    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:01:43.053991 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:43.053960    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:01:45.926409 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:45.925514    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:01:45.926409 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:45.925617    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:01:50.467755 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:50.467721    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:01:59.010033 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:01:59.010003    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:02:05.147535 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:02:05.147501    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:02:07.776407 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:02:07.776360    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:02:27.146164 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:02:27.146127    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:02:35.148425 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:02:35.148375    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:02:43.310679 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:02:43.310650    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 16:02:43.317440 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:02:43.317420    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 16:02:56.166546 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:02:56.166465    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:03:05.147777 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:03:05.147736    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:03:17.532661 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:17.532627    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:03:28.248084 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:28.248052    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:03:35.147540 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:03:35.147496    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:03:48.408585 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.408555    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"]
Nov 01 16:03:48.409118 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.408933    5423 cpu_manager.go:414] "containerMap: removing container" podUID="1314115e-f7e4-4c2e-bf24-1d5602d3fedb" containerName="payload-processing"
Nov 01 16:03:48.409118 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.408944    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1314115e-f7e4-4c2e-bf24-1d5602d3fedb" containerName="payload-processing"
Nov 01 16:03:48.409118 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.409025    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="1314115e-f7e4-4c2e-bf24-1d5602d3fedb" containerName="payload-processing"
Nov 01 16:03:48.412112 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.412092    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.515005 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.514972    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"]
Nov 01 16:03:48.534200 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534167    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-data\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.534381 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534234    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gds\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-kube-api-access-f8gds\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.534483 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534348    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-credential-socket\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.534558 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534486    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-token\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.534558 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534528    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-certs\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.534699 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534558    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-envoy\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.534699 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534622    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-podinfo\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.534699 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534651    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-socket\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.534699 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.534678    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3e60-d2dd-4a1c-b638-964a50c99579-istiod-ca-cert\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.564719 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.564693    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-9ebec3-openshift-default-dockercfg-56qcz\""
Nov 01 16:03:48.635142 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635110    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-socket\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635273 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635148    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3e60-d2dd-4a1c-b638-964a50c99579-istiod-ca-cert\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635273 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635223    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-data\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635376 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635270    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gds\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-kube-api-access-f8gds\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635376 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635347    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-credential-socket\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635515 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635382    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-token\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635515 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635425    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-certs\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635515 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635459    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-envoy\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635672 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635521    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-podinfo\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.635672 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.635642    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-socket\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.636406 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.636359    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-data\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.636600 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.636503    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3e60-d2dd-4a1c-b638-964a50c99579-istiod-ca-cert\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.636717 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.636654    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-certs\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.636835 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.636811    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-credential-socket\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.638822 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.638788    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-envoy\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.638946 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.638930    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-podinfo\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.694207 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.694127    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-token\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.694379 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.694261    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gds\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-kube-api-access-f8gds\") pod \"e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") " pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:48.725002 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:48.724971    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:49.206438 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:49.206417    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"]
Nov 01 16:03:49.208574 ip-10-0-135-142 kubenswrapper[5423]: W1101 16:03:49.208536    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ae3e60_d2dd_4a1c_b638_964a50c99579.slice/crio-18dd93e5ba090d02ffd634e6802c1e91deed73921074680a4196cb2a5cd7dd65 WatchSource:0}: Error finding container 18dd93e5ba090d02ffd634e6802c1e91deed73921074680a4196cb2a5cd7dd65: Status 404 returned error can't find the container with id 18dd93e5ba090d02ffd634e6802c1e91deed73921074680a4196cb2a5cd7dd65
Nov 01 16:03:49.210968 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:49.210938    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 16:03:49.211084 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:49.211002    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 16:03:49.211084 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:49.211039    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 16:03:49.400851 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:49.400811    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" event={"ID":"28ae3e60-d2dd-4a1c-b638-964a50c99579","Type":"ContainerStarted","Data":"1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86"}
Nov 01 16:03:49.401048 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:49.400890    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" event={"ID":"28ae3e60-d2dd-4a1c-b638-964a50c99579","Type":"ContainerStarted","Data":"18dd93e5ba090d02ffd634e6802c1e91deed73921074680a4196cb2a5cd7dd65"}
Nov 01 16:03:49.519650 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:49.519548    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" podStartSLOduration=1.519534432 podStartE2EDuration="1.519534432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:49.519163004 +0000 UTC m=+6369.684648304" watchObservedRunningTime="2026-11-01 16:03:49.519534432 +0000 UTC m=+6369.685019729"
Nov 01 16:03:49.725420 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:49.725366    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:50.730954 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:50.730925    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:51.409980 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:51.409949    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:51.410885 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:51.410865    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:03:57.845485 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:57.845453    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"]
Nov 01 16:03:57.849232 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:57.849214    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:57.904209 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:57.904178    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-9ebec3-serving-cert\""
Nov 01 16:03:57.906219 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:57.906186    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"]
Nov 01 16:03:57.922779 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:57.922748    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmgrf\" (UniqueName: \"kubernetes.io/projected/c857c3a6-656d-421d-8a15-0978656e31a3-kube-api-access-hmgrf\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:57.922779 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:57.922780    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:57.923009 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:57.922934    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:57.923009 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:57.922974    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c857c3a6-656d-421d-8a15-0978656e31a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.023720 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.023679    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.023720 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.023720    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c857c3a6-656d-421d-8a15-0978656e31a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.023942 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.023801    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmgrf\" (UniqueName: \"kubernetes.io/projected/c857c3a6-656d-421d-8a15-0978656e31a3-kube-api-access-hmgrf\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.023942 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.023818    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.023942 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:03:58.023936    5423 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-serving-cert: secret "maas-api-e2e-worker-single-main-9ebec3-serving-cert" not found
Nov 01 16:03:58.024086 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:03:58.024021    5423 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls podName:c857c3a6-656d-421d-8a15-0978656e31a3 nodeName:}" failed. No retries permitted until 2026-11-01 16:03:58.523998245 +0000 UTC m=+6378.689483524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls") pod "maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z" (UID: "c857c3a6-656d-421d-8a15-0978656e31a3") : secret "maas-api-e2e-worker-single-main-9ebec3-serving-cert" not found
Nov 01 16:03:58.024493 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.024473    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c857c3a6-656d-421d-8a15-0978656e31a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.026130 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.026107    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.089248 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.089209    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmgrf\" (UniqueName: \"kubernetes.io/projected/c857c3a6-656d-421d-8a15-0978656e31a3-kube-api-access-hmgrf\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.540838 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.540801    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.543535 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.543503    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:58.762788 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:58.762752    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:03:59.133812 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:59.133789    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"]
Nov 01 16:03:59.135905 ip-10-0-135-142 kubenswrapper[5423]: W1101 16:03:59.135878    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc857c3a6_656d_421d_8a15_0978656e31a3.slice/crio-cfb5729e847e949208e56a2bb6510a59cbc55f8889094177e733c4621017a2e3 WatchSource:0}: Error finding container cfb5729e847e949208e56a2bb6510a59cbc55f8889094177e733c4621017a2e3: Status 404 returned error can't find the container with id cfb5729e847e949208e56a2bb6510a59cbc55f8889094177e733c4621017a2e3
Nov 01 16:03:59.137490 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:59.137468    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 16:03:59.449149 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:03:59.449069    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z" event={"ID":"c857c3a6-656d-421d-8a15-0978656e31a3","Type":"ContainerStarted","Data":"cfb5729e847e949208e56a2bb6510a59cbc55f8889094177e733c4621017a2e3"}
Nov 01 16:04:01.459454 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:01.459415    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z" event={"ID":"c857c3a6-656d-421d-8a15-0978656e31a3","Type":"ContainerStarted","Data":"946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146"}
Nov 01 16:04:01.459953 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:01.459610    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:04:01.575325 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:01.575267    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z" podStartSLOduration=3.090340059 podStartE2EDuration="4.575253925s" totalImagesPullingTime="1.484913866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:01.575172549 +0000 UTC m=+6381.740657846" watchObservedRunningTime="2026-11-01 16:04:01.575253925 +0000 UTC m=+6381.740739222"
Nov 01 16:04:05.150544 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:04:05.150500    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:04:05.275668 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:05.275631    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:04:07.469805 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:07.469772    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:04:14.550486 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:14.550449    5423 generic.go:356] "Generic (PLEG): container finished" podID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerID="1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86" exitCode=0
Nov 01 16:04:14.551023 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:14.550525    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" event={"ID":"28ae3e60-d2dd-4a1c-b638-964a50c99579","Type":"ContainerDied","Data":"1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86"}
Nov 01 16:04:14.551102 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:14.551027    5423 scope.go:122] "RemoveContainer" containerID="1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86"
Nov 01 16:04:14.552242 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:14.552216    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 16:04:14.552357 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:14.552280    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 16:04:14.552357 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:14.552315    5423 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 16:04:15.556817 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:15.556782    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" event={"ID":"28ae3e60-d2dd-4a1c-b638-964a50c99579","Type":"ContainerStarted","Data":"97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce"}
Nov 01 16:04:15.725411 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:15.725347    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:04:15.725610 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:15.725557    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 01 16:04:15.725676 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:15.725601    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 01 16:04:16.725865 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:16.725821    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 01 16:04:16.726280 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:16.725876    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 01 16:04:17.725625 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:17.725589    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 01 16:04:17.725819 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:17.725648    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 01 16:04:18.726027 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:18.725994    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 01 16:04:18.726480 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:18.726045    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 01 16:04:19.725442 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:19.725350    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 01 16:04:19.725685 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:19.725436    5423 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 01 16:04:20.730323 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:20.730294    5423 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:04:20.730751 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:20.730576    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:04:20.731295 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:20.731275    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:04:25.466444 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:25.466410    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:04:35.149057 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:04:35.149020    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:04:50.387161 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:04:50.387119    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:05:05.148203 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:05:05.148167    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:05:19.347120 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:19.347081    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:05:26.916189 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:26.916153    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"]
Nov 01 16:05:26.916751 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:26.916375    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z" podUID="c857c3a6-656d-421d-8a15-0978656e31a3" containerName="maas-api" containerID="cri-o://946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146" gracePeriod=30
Nov 01 16:05:27.465401 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:27.465358    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z" podUID="c857c3a6-656d-421d-8a15-0978656e31a3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.7:8443/health\": dial tcp 10.133.1.7:8443: connect: connection refused"
Nov 01 16:05:28.168804 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.168780    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:05:28.338609 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.338580    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-metrics-certs\") pod \"c857c3a6-656d-421d-8a15-0978656e31a3\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") "
Nov 01 16:05:28.338773 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.338627    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c857c3a6-656d-421d-8a15-0978656e31a3-kube-api-access-hmgrf\" (UniqueName: \"kubernetes.io/projected/c857c3a6-656d-421d-8a15-0978656e31a3-kube-api-access-hmgrf\") pod \"c857c3a6-656d-421d-8a15-0978656e31a3\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") "
Nov 01 16:05:28.338773 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.338723    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls\") pod \"c857c3a6-656d-421d-8a15-0978656e31a3\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") "
Nov 01 16:05:28.338869 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.338796    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c857c3a6-656d-421d-8a15-0978656e31a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c857c3a6-656d-421d-8a15-0978656e31a3-openshift-service-ca\") pod \"c857c3a6-656d-421d-8a15-0978656e31a3\" (UID: \"c857c3a6-656d-421d-8a15-0978656e31a3\") "
Nov 01 16:05:28.339197 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.339172    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c857c3a6-656d-421d-8a15-0978656e31a3-openshift-service-ca" pod "c857c3a6-656d-421d-8a15-0978656e31a3" (UID: "c857c3a6-656d-421d-8a15-0978656e31a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 16:05:28.340566 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.340547    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-metrics-certs" pod "c857c3a6-656d-421d-8a15-0978656e31a3" (UID: "c857c3a6-656d-421d-8a15-0978656e31a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 16:05:28.341145 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.341122    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls" pod "c857c3a6-656d-421d-8a15-0978656e31a3" (UID: "c857c3a6-656d-421d-8a15-0978656e31a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 16:05:28.341202 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.341146    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c857c3a6-656d-421d-8a15-0978656e31a3-kube-api-access-hmgrf" pod "c857c3a6-656d-421d-8a15-0978656e31a3" (UID: "c857c3a6-656d-421d-8a15-0978656e31a3"). InnerVolumeSpecName "kube-api-access-hmgrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 16:05:28.440031 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.440000    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c857c3a6-656d-421d-8a15-0978656e31a3-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:28.440031 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.440029    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:28.440031 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.440039    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmgrf\" (UniqueName: \"kubernetes.io/projected/c857c3a6-656d-421d-8a15-0978656e31a3-kube-api-access-hmgrf\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:28.440270 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.440048    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c857c3a6-656d-421d-8a15-0978656e31a3-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:28.896138 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.896102    5423 generic.go:356] "Generic (PLEG): container finished" podID="c857c3a6-656d-421d-8a15-0978656e31a3" containerID="946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146" exitCode=0
Nov 01 16:05:28.896332 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.896144    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"
Nov 01 16:05:28.896332 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.896181    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z" event={"ID":"c857c3a6-656d-421d-8a15-0978656e31a3","Type":"ContainerDied","Data":"946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146"}
Nov 01 16:05:28.896332 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.896233    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z" event={"ID":"c857c3a6-656d-421d-8a15-0978656e31a3","Type":"ContainerDied","Data":"cfb5729e847e949208e56a2bb6510a59cbc55f8889094177e733c4621017a2e3"}
Nov 01 16:05:28.896332 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.896258    5423 scope.go:122] "RemoveContainer" containerID="946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146"
Nov 01 16:05:28.905544 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.905523    5423 scope.go:122] "RemoveContainer" containerID="946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146"
Nov 01 16:05:28.905818 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:05:28.905800    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146\": container with ID starting with 946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146 not found: ID does not exist" containerID="946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146"
Nov 01 16:05:28.905867 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:28.905827    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146"} err="failed to get container status \"946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146\": rpc error: code = NotFound desc = could not find container \"946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146\": container with ID starting with 946e3e665cd20eded9805f11f5a1d36e8b92e15991f4f7485c6c3c1f64f3a146 not found: ID does not exist"
Nov 01 16:05:29.065231 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:29.065197    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"]
Nov 01 16:05:29.116290 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:29.116256    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9ebec3-86f7c8fd6c-44g8z"]
Nov 01 16:05:29.959180 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:29.959145    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c857c3a6-656d-421d-8a15-0978656e31a3" path="/var/lib/kubelet/pods/c857c3a6-656d-421d-8a15-0978656e31a3/volumes"
Nov 01 16:05:30.647043 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:05:30.647005    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:05:35.148358 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:05:35.148318    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:06:01.032369 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:01.032291    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"]
Nov 01 16:06:01.032956 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:01.032547    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy" containerID="cri-o://97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce" gracePeriod=30
Nov 01 16:06:05.148944 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:06:05.148901    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:06:05.733142 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:05.733108    5423 patch_prober.go:29] interesting pod/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 16:06:05.733323 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:05.733170    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 16:06:06.291969 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.291945    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:06:06.386922 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.386895    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-socket\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387075 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.386937    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-kube-api-access-f8gds\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-kube-api-access-f8gds\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387075 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.386967    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-token\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387075 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.386999    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-certs\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387075 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387047    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ae3e60-d2dd-4a1c-b638-964a50c99579-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3e60-d2dd-4a1c-b638-964a50c99579-istiod-ca-cert\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387312 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387169    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-socket" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:06:06.387312 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387190    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-credential-socket\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387312 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387273    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-envoy\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387513 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387312    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-data\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387513 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387317    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-certs" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:06:06.387513 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387446    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-credential-socket" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:06:06.387513 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387458    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ae3e60-d2dd-4a1c-b638-964a50c99579-istiod-ca-cert" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 16:06:06.387513 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387456    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-podinfo\") pod \"28ae3e60-d2dd-4a1c-b638-964a50c99579\" (UID: \"28ae3e60-d2dd-4a1c-b638-964a50c99579\") "
Nov 01 16:06:06.387770 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387610    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-data" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:06:06.387942 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387920    5423 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:06.387994 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387949    5423 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-workload-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:06.387994 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387966    5423 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3e60-d2dd-4a1c-b638-964a50c99579-istiod-ca-cert\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:06.387994 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387982    5423 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-credential-socket\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:06.388105 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.387997    5423 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-data\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:06.389211 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.389187    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-token" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 16:06:06.389321 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.389303    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-podinfo" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 16:06:06.389600 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.389578    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-envoy" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:06:06.389600 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.389591    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-kube-api-access-f8gds" pod "28ae3e60-d2dd-4a1c-b638-964a50c99579" (UID: "28ae3e60-d2dd-4a1c-b638-964a50c99579"). InnerVolumeSpecName "kube-api-access-f8gds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 16:06:06.488795 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.488767    5423 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-envoy\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:06.488795 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.488792    5423 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-podinfo\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:06.488943 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.488803    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8gds\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-kube-api-access-f8gds\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:06.488943 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:06.488813    5423 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3e60-d2dd-4a1c-b638-964a50c99579-istio-token\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:07.063091 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.063053    5423 generic.go:356] "Generic (PLEG): container finished" podID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerID="97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce" exitCode=0
Nov 01 16:06:07.063269 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.063129    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"
Nov 01 16:06:07.063269 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.063129    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" event={"ID":"28ae3e60-d2dd-4a1c-b638-964a50c99579","Type":"ContainerDied","Data":"97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce"}
Nov 01 16:06:07.063269 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.063182    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd" event={"ID":"28ae3e60-d2dd-4a1c-b638-964a50c99579","Type":"ContainerDied","Data":"18dd93e5ba090d02ffd634e6802c1e91deed73921074680a4196cb2a5cd7dd65"}
Nov 01 16:06:07.063269 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.063202    5423 scope.go:122] "RemoveContainer" containerID="97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce"
Nov 01 16:06:07.072177 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.072160    5423 scope.go:122] "RemoveContainer" containerID="1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86"
Nov 01 16:06:07.079579 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.079560    5423 scope.go:122] "RemoveContainer" containerID="97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce"
Nov 01 16:06:07.079834 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:06:07.079815    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce\": container with ID starting with 97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce not found: ID does not exist" containerID="97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce"
Nov 01 16:06:07.079884 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.079843    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce"} err="failed to get container status \"97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce\": rpc error: code = NotFound desc = could not find container \"97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce\": container with ID starting with 97d2dece30eb601c196eaf3b644bbba3d282ead142caff2286a7ab65cacfc1ce not found: ID does not exist"
Nov 01 16:06:07.079884 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.079860    5423 scope.go:122] "RemoveContainer" containerID="1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86"
Nov 01 16:06:07.080117 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:06:07.080099    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86\": container with ID starting with 1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86 not found: ID does not exist" containerID="1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86"
Nov 01 16:06:07.080160 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.080124    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86"} err="failed to get container status \"1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86\": rpc error: code = NotFound desc = could not find container \"1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86\": container with ID starting with 1b43c4a84d47b93747bd4e113d5c6641fa5606c42a48276fe2c441ad11cd4e86 not found: ID does not exist"
Nov 01 16:06:07.280840 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.280792    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"]
Nov 01 16:06:07.331831 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.331761    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9ebec3-openshift-default-67fb4f6fb5rr4nd"]
Nov 01 16:06:07.958909 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:06:07.958873    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" path="/var/lib/kubelet/pods/28ae3e60-d2dd-4a1c-b638-964a50c99579/volumes"
Nov 01 16:06:35.148075 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:06:35.148038    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:07:05.148331 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:07:05.148293    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:07:35.149204 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:07:35.149115    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:07:43.341484 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:07:43.341458    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 16:07:43.345347 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:07:43.345325    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 16:07:49.934622 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:07:49.934586    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:08:05.150216 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:08:05.150175    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:08:06.841016 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:06.840975    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:08:15.533833 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:15.533793    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:08:23.333858 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:23.333818    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:08:28.106112 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106075    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"]
Nov 01 16:08:28.106564 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106491    5423 cpu_manager.go:414] "containerMap: removing container" podUID="c857c3a6-656d-421d-8a15-0978656e31a3" containerName="maas-api"
Nov 01 16:08:28.106564 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106502    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c857c3a6-656d-421d-8a15-0978656e31a3" containerName="maas-api"
Nov 01 16:08:28.106564 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106514    5423 cpu_manager.go:414] "containerMap: removing container" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy"
Nov 01 16:08:28.106564 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106521    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy"
Nov 01 16:08:28.106564 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106533    5423 cpu_manager.go:414] "containerMap: removing container" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy"
Nov 01 16:08:28.106564 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106539    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy"
Nov 01 16:08:28.106765 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106599    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy"
Nov 01 16:08:28.106765 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.106614    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="c857c3a6-656d-421d-8a15-0978656e31a3" containerName="maas-api"
Nov 01 16:08:28.109998 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.109973    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.215748 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.215705    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"]
Nov 01 16:08:28.250986 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.250949    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-metrics-certs\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.250986 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.250989    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-maas-api-tls\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.251229 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.251159    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-openshift-service-ca\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.251229 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.251212    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cx4\" (UniqueName: \"kubernetes.io/projected/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-kube-api-access-x7cx4\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.352649 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.352607    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-metrics-certs\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.352859 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.352660    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-maas-api-tls\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.352859 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.352780    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-openshift-service-ca\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.352859 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.352840    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cx4\" (UniqueName: \"kubernetes.io/projected/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-kube-api-access-x7cx4\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.353507 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.353484    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-openshift-service-ca\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.355330 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.355308    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-metrics-certs\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.355449 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.355377    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-maas-api-tls\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.410755 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.410680    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cx4\" (UniqueName: \"kubernetes.io/projected/a7d6e24b-ba4c-4ddf-9206-37eee97f52bc-kube-api-access-x7cx4\") pod \"maas-api-58788d547c-s8vjg\" (UID: \"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.424524 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.424486    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:28.800004 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:28.799914    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"]
Nov 01 16:08:28.803211 ip-10-0-135-142 kubenswrapper[5423]: W1101 16:08:28.803177    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d6e24b_ba4c_4ddf_9206_37eee97f52bc.slice/crio-d30b58d810ab868cba5ba747b2dcd29c5f4649d92db0852bdaaac5cb1a83c4f1 WatchSource:0}: Error finding container d30b58d810ab868cba5ba747b2dcd29c5f4649d92db0852bdaaac5cb1a83c4f1: Status 404 returned error can't find the container with id d30b58d810ab868cba5ba747b2dcd29c5f4649d92db0852bdaaac5cb1a83c4f1
Nov 01 16:08:29.748535 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:29.748495    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg" event={"ID":"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc","Type":"ContainerStarted","Data":"d30b58d810ab868cba5ba747b2dcd29c5f4649d92db0852bdaaac5cb1a83c4f1"}
Nov 01 16:08:30.755108 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:30.755060    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg" event={"ID":"a7d6e24b-ba4c-4ddf-9206-37eee97f52bc","Type":"ContainerStarted","Data":"ac998ca99437dc57f576a437bc76b69bcd37889396421cf06f86c94a2323fa02"}
Nov 01 16:08:30.755584 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:30.755152    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:30.871702 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:30.871639    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg" podStartSLOduration=1.933292782 podStartE2EDuration="2.87162281s" totalImagesPullingTime="938.330028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:30.870433381 +0000 UTC m=+6651.035918676" watchObservedRunningTime="2026-11-01 16:08:30.87162281 +0000 UTC m=+6651.037108105"
Nov 01 16:08:35.150887 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:08:35.150841    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:08:36.766276 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:36.766243    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58788d547c-s8vjg"
Nov 01 16:08:36.937184 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:36.937145    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"]
Nov 01 16:08:36.937477 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:36.937429    5423 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x" podUID="c53e3e14-ab21-43c8-b7d6-8cb588674e12" containerName="maas-api" containerID="cri-o://14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6" gracePeriod=30
Nov 01 16:08:38.194435 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.194400    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 16:08:38.354165 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.354130    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53e3e14-ab21-43c8-b7d6-8cb588674e12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53e3e14-ab21-43c8-b7d6-8cb588674e12-openshift-service-ca\") pod \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") "
Nov 01 16:08:38.354411 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.354195    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-maas-api-tls\") pod \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") "
Nov 01 16:08:38.354411 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.354301    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-metrics-certs\") pod \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") "
Nov 01 16:08:38.354411 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.354339    5423 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53e3e14-ab21-43c8-b7d6-8cb588674e12-kube-api-access-qs89n\" (UniqueName: \"kubernetes.io/projected/c53e3e14-ab21-43c8-b7d6-8cb588674e12-kube-api-access-qs89n\") pod \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\" (UID: \"c53e3e14-ab21-43c8-b7d6-8cb588674e12\") "
Nov 01 16:08:38.354684 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.354557    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53e3e14-ab21-43c8-b7d6-8cb588674e12-openshift-service-ca" pod "c53e3e14-ab21-43c8-b7d6-8cb588674e12" (UID: "c53e3e14-ab21-43c8-b7d6-8cb588674e12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 16:08:38.354684 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.354670    5423 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53e3e14-ab21-43c8-b7d6-8cb588674e12-openshift-service-ca\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:08:38.356594 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.356561    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-maas-api-tls" pod "c53e3e14-ab21-43c8-b7d6-8cb588674e12" (UID: "c53e3e14-ab21-43c8-b7d6-8cb588674e12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 16:08:38.356594 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.356579    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53e3e14-ab21-43c8-b7d6-8cb588674e12-kube-api-access-qs89n" pod "c53e3e14-ab21-43c8-b7d6-8cb588674e12" (UID: "c53e3e14-ab21-43c8-b7d6-8cb588674e12"). InnerVolumeSpecName "kube-api-access-qs89n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 16:08:38.356745 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.356609    5423 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-metrics-certs" pod "c53e3e14-ab21-43c8-b7d6-8cb588674e12" (UID: "c53e3e14-ab21-43c8-b7d6-8cb588674e12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 16:08:38.457329 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.457291    5423 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-maas-api-tls\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:08:38.457329 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.457326    5423 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e3e14-ab21-43c8-b7d6-8cb588674e12-metrics-certs\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:08:38.457596 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.457340    5423 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs89n\" (UniqueName: \"kubernetes.io/projected/c53e3e14-ab21-43c8-b7d6-8cb588674e12-kube-api-access-qs89n\") on node \"ip-10-0-135-142.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:08:38.794892 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.794805    5423 generic.go:356] "Generic (PLEG): container finished" podID="c53e3e14-ab21-43c8-b7d6-8cb588674e12" containerID="14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6" exitCode=0
Nov 01 16:08:38.794892 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.794845    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x" event={"ID":"c53e3e14-ab21-43c8-b7d6-8cb588674e12","Type":"ContainerDied","Data":"14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6"}
Nov 01 16:08:38.795102 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.794912    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x" event={"ID":"c53e3e14-ab21-43c8-b7d6-8cb588674e12","Type":"ContainerDied","Data":"a3b12983897e7e1990cb9e40525c5122e00fd132521d3a8c5c1c7c94911f44cc"}
Nov 01 16:08:38.795102 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.794936    5423 scope.go:122] "RemoveContainer" containerID="14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6"
Nov 01 16:08:38.795102 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.794859    5423 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"
Nov 01 16:08:38.805879 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.805858    5423 scope.go:122] "RemoveContainer" containerID="14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6"
Nov 01 16:08:38.806187 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:08:38.806165    5423 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6\": container with ID starting with 14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6 not found: ID does not exist" containerID="14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6"
Nov 01 16:08:38.806236 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.806198    5423 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6"} err="failed to get container status \"14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6\": rpc error: code = NotFound desc = could not find container \"14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6\": container with ID starting with 14b84635d846502a8d5710cc55698f22b3b6d614feda5e1c105b5e892b0d50c6 not found: ID does not exist"
Nov 01 16:08:38.960304 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:38.960267    5423 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"]
Nov 01 16:08:39.011157 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:39.011120    5423 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-6f24x"]
Nov 01 16:08:39.958567 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:39.958534    5423 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53e3e14-ab21-43c8-b7d6-8cb588674e12" path="/var/lib/kubelet/pods/c53e3e14-ab21-43c8-b7d6-8cb588674e12/volumes"
Nov 01 16:08:40.358088 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:08:40.358049    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:09:05.148900 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:09:05.148858    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:09:16.753472 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:09:16.753424    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:09:28.457914 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:09:28.457878    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:09:35.149103 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:09:35.149064    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:09:49.859422 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:09:49.859373    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:10:01.363433 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:10:01.363397    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:10:05.148695 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:10:05.148653    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:10:32.060141 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:10:32.060044    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:10:35.149298 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:10:35.149259    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:10:42.739942 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:10:42.739905    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:10:54.579625 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:10:54.579585    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:10:57.059626 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:10:57.059581    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:10:57.060274 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:10:57.060234    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:11:04.879366 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:11:04.879329    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:11:05.147851 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:11:05.147772    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:11:28.266180 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:11:28.266140    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:11:30.917412 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:11:30.914511    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wfr6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:11:30.917412 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:11:30.914607    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wfr6h" podUID="d0dd2f89-8808-4b54-8255-e0c4dbd31fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:11:35.148423 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:11:35.148364    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:11:39.035610 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:11:39.035571    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:11:48.284603 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:11:48.284569    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:12:05.148964 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:12:05.148923    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:12:07.370261 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:12:07.370225    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:12:09.928572 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:12:09.928509    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gkt8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:12:09.929277 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:12:09.929227    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gkt8b" podUID="35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:12:17.359974 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:12:17.359939    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:12:35.151042 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:12:35.150998    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:12:43.372110 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:12:43.372082    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 16:12:43.373004 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:12:43.372987    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k8d4s_08c55c12-d679-4874-9e6f-789e7829641c/ovn-acl-logging/0.log"
Nov 01 16:13:05.150746 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:13:05.148429    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:13:35.147709 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:13:35.147629    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:14:05.259410 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:14:05.245573    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:14:25.243706 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:14:25.243667    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dkfnx"]
Nov 01 16:14:27.060544 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:14:27.060498    5423 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pw4lq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:14:27.061171 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:14:27.061135    5423 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pw4lq" podUID="c3e03db8-a460-41e6-8eb4-a044121bd5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:14:35.148867 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:14:35.148831    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:14:40.301655 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:14:40.301621    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d985f759b-hcptd_590e6cf3-e96c-4b31-a570-555017c2bb47/authorino/0.log"
Nov 01 16:14:51.574873 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:14:51.574792    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9f778ddb6-jnqbk_0afcba4a-013c-42bd-8216-b9b91a2c4bde/manager/0.log"
Nov 01 16:14:52.996072 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:14:52.996045    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75bf7b5bf6-4vbfk_1c0d579d-61a9-4213-a72d-dcf2a340496b/manager/0.log"
Nov 01 16:14:55.444948 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:14:55.444916    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58788d547c-s8vjg_a7d6e24b-ba4c-4ddf-9206-37eee97f52bc/maas-api/0.log"
Nov 01 16:15:01.123008 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:01.122972    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d985f759b-hcptd_590e6cf3-e96c-4b31-a570-555017c2bb47/authorino/0.log"
Nov 01 16:15:02.004298 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:02.004268    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-55crp_f521cd06-308b-48bb-a9a3-1ee4a82fbe04/kuadrant-console-plugin/0.log"
Nov 01 16:15:02.831349 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:02.831325    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dkfnx_a4d27b75-03ea-410d-b323-86daa17409e9/limitador/0.log"
Nov 01 16:15:04.371084 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:04.371051    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qk6ww_3cb347b5-2346-4750-bf81-d005c9ae3fd9/discovery/0.log"
Nov 01 16:15:04.934628 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:04.934590    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fc9ccddb-v89s4_096fc6d0-c03c-4cb9-b330-3a71744e4470/kube-auth-proxy/0.log"
Nov 01 16:15:05.151836 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:15:05.151789    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:15:05.993751 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:05.993721    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-gkt8b_35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7/istio-proxy/0.log"
Nov 01 16:15:06.859978 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:06.859950    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pw4lq_c3e03db8-a460-41e6-8eb4-a044121bd5dc/istio-proxy/0.log"
Nov 01 16:15:07.470454 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:07.470425    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wfr6h_d0dd2f89-8808-4b54-8255-e0c4dbd31fe9/istio-proxy/0.log"
Nov 01 16:15:08.325010 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:08.324980    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-688d45fc-hm5jh_ae261db3-a26e-4f0b-ad85-6a487a3faeda/payload-pre-processing/0.log"
Nov 01 16:15:08.544237 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:08.544210    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-dd858cb66-68dsg_2678e74b-8388-4f25-b832-934ccd98d338/payload-processing/0.log"
Nov 01 16:15:10.086606 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:10.086580    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l2q7z_635d4701-f7ba-4bbc-81bc-352f89a2f4f3/main/0.log"
Nov 01 16:15:10.564260 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:10.564232    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-hxdbf_946e8442-e532-48b6-a5ad-ef769fb2e71c/main/0.log"
Nov 01 16:15:10.986235 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:10.986164    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wdlwv_beffa1b9-f1df-4269-a237-322aaecf3c18/main/0.log"
Nov 01 16:15:11.528284 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:11.528258    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gknfk_ad617adb-5b9f-4927-ad5e-b8083aa5d1c2/main/0.log"
Nov 01 16:15:21.127916 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.127882    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8k5v4/must-gather-p76cz"]
Nov 01 16:15:21.128306 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.128283    5423 cpu_manager.go:414] "containerMap: removing container" podUID="c53e3e14-ab21-43c8-b7d6-8cb588674e12" containerName="maas-api"
Nov 01 16:15:21.128306 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.128297    5423 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53e3e14-ab21-43c8-b7d6-8cb588674e12" containerName="maas-api"
Nov 01 16:15:21.128380 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.128365    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ae3e60-d2dd-4a1c-b638-964a50c99579" containerName="istio-proxy"
Nov 01 16:15:21.128380 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.128376    5423 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53e3e14-ab21-43c8-b7d6-8cb588674e12" containerName="maas-api"
Nov 01 16:15:21.131483 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.131464    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k5v4/must-gather-p76cz"
Nov 01 16:15:21.184741 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.184708    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k5v4\"/\"kube-root-ca.crt\""
Nov 01 16:15:21.225541 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.225513    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e1114f6-6d64-4039-a87d-a8f469d9fe64-must-gather-output\") pod \"must-gather-p76cz\" (UID: \"2e1114f6-6d64-4039-a87d-a8f469d9fe64\") " pod="openshift-must-gather-8k5v4/must-gather-p76cz"
Nov 01 16:15:21.225714 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.225570    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk27t\" (UniqueName: \"kubernetes.io/projected/2e1114f6-6d64-4039-a87d-a8f469d9fe64-kube-api-access-mk27t\") pod \"must-gather-p76cz\" (UID: \"2e1114f6-6d64-4039-a87d-a8f469d9fe64\") " pod="openshift-must-gather-8k5v4/must-gather-p76cz"
Nov 01 16:15:21.235828 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.235804    5423 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k5v4\"/\"openshift-service-ca.crt\""
Nov 01 16:15:21.235968 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.235882    5423 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8k5v4\"/\"default-dockercfg-tf4nc\""
Nov 01 16:15:21.237576 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.237556    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k5v4/must-gather-p76cz"]
Nov 01 16:15:21.326857 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.326829    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk27t\" (UniqueName: \"kubernetes.io/projected/2e1114f6-6d64-4039-a87d-a8f469d9fe64-kube-api-access-mk27t\") pod \"must-gather-p76cz\" (UID: \"2e1114f6-6d64-4039-a87d-a8f469d9fe64\") " pod="openshift-must-gather-8k5v4/must-gather-p76cz"
Nov 01 16:15:21.327019 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.326975    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e1114f6-6d64-4039-a87d-a8f469d9fe64-must-gather-output\") pod \"must-gather-p76cz\" (UID: \"2e1114f6-6d64-4039-a87d-a8f469d9fe64\") " pod="openshift-must-gather-8k5v4/must-gather-p76cz"
Nov 01 16:15:21.327327 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.327310    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2e1114f6-6d64-4039-a87d-a8f469d9fe64-must-gather-output\") pod \"must-gather-p76cz\" (UID: \"2e1114f6-6d64-4039-a87d-a8f469d9fe64\") " pod="openshift-must-gather-8k5v4/must-gather-p76cz"
Nov 01 16:15:21.384917 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.384847    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk27t\" (UniqueName: \"kubernetes.io/projected/2e1114f6-6d64-4039-a87d-a8f469d9fe64-kube-api-access-mk27t\") pod \"must-gather-p76cz\" (UID: \"2e1114f6-6d64-4039-a87d-a8f469d9fe64\") " pod="openshift-must-gather-8k5v4/must-gather-p76cz"
Nov 01 16:15:21.444890 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.444858    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k5v4/must-gather-p76cz"
Nov 01 16:15:21.818774 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.818684    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k5v4/must-gather-p76cz"]
Nov 01 16:15:21.821071 ip-10-0-135-142 kubenswrapper[5423]: W1101 16:15:21.821050    5423 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e1114f6_6d64_4039_a87d_a8f469d9fe64.slice/crio-c3153145fe2c45aa1d1c623be9720d847fccfacd3473672c8581d5ab1f68cd25 WatchSource:0}: Error finding container c3153145fe2c45aa1d1c623be9720d847fccfacd3473672c8581d5ab1f68cd25: Status 404 returned error can't find the container with id c3153145fe2c45aa1d1c623be9720d847fccfacd3473672c8581d5ab1f68cd25
Nov 01 16:15:21.822864 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:21.822845    5423 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 16:15:22.028823 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:22.028789    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/must-gather-p76cz" event={"ID":"2e1114f6-6d64-4039-a87d-a8f469d9fe64","Type":"ContainerStarted","Data":"c3153145fe2c45aa1d1c623be9720d847fccfacd3473672c8581d5ab1f68cd25"}
Nov 01 16:15:24.040672 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:24.040631    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/must-gather-p76cz" event={"ID":"2e1114f6-6d64-4039-a87d-a8f469d9fe64","Type":"ContainerStarted","Data":"f1080ef8508c856d940614a61ebf4a01d66d5f1ab6356dc16b0babc06ece1c7c"}
Nov 01 16:15:24.041144 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:24.040680    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/must-gather-p76cz" event={"ID":"2e1114f6-6d64-4039-a87d-a8f469d9fe64","Type":"ContainerStarted","Data":"af8ea2ad2844fc450a19ab640dcc421c30654c85b4d2ccacd9a11f9ebe2391c1"}
Nov 01 16:15:24.155908 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:24.155854    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8k5v4/must-gather-p76cz" podStartSLOduration=1.725177008 podStartE2EDuration="3.155841632s" totalImagesPullingTime="1.430664624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:24.155376144 +0000 UTC m=+7064.320861441" watchObservedRunningTime="2026-11-01 16:15:24.155841632 +0000 UTC m=+7064.321326929"
Nov 01 16:15:31.556406 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:31.553049    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z9bf5_87bcc491-1e33-442b-b230-0cb7ae30f544/global-pull-secret-syncer/0.log"
Nov 01 16:15:33.993428 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:33.992776    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g44v5_120f88bc-d7ec-43a0-8503-726fb3ca3947/konnectivity-agent/0.log"
Nov 01 16:15:34.987760 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:34.987722    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9f778ddb6-jnqbk_0afcba4a-013c-42bd-8216-b9b91a2c4bde/manager/0.log"
Nov 01 16:15:35.171222 ip-10-0-135-142 kubenswrapper[5423]: E1101 16:15:35.171180    5423 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq7mp" containerName="extractor"
Nov 01 16:15:35.746668 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:35.746633    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-142.us-west-2.compute.internal_fccbcd60b09b5d4721c0b990d6d1ecb1/haproxy/0.log"
Nov 01 16:15:36.135494 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:36.135460    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75bf7b5bf6-4vbfk_1c0d579d-61a9-4213-a72d-dcf2a340496b/manager/0.log"
Nov 01 16:15:40.022635 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:40.022598    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d985f759b-hcptd_590e6cf3-e96c-4b31-a570-555017c2bb47/authorino/0.log"
Nov 01 16:15:40.969515 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:40.969472    5423 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"]
Nov 01 16:15:40.977571 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:40.977538    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.028256 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.028216    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"]
Nov 01 16:15:41.056430 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.051970    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-lib-modules\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.056430 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.052058    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lm25\" (UniqueName: \"kubernetes.io/projected/025373f7-7969-47b6-9fe8-ee0b49d5bd74-kube-api-access-8lm25\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.056430 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.052091    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-proc\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.056430 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.052182    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-podres\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.056430 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.052240    5423 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-sys\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.148666 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.148630    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9f778ddb6-jnqbk_0afcba4a-013c-42bd-8216-b9b91a2c4bde/manager/0.log"
Nov 01 16:15:41.152847 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.152804    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-podres\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.153043 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.152884    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-sys\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.153043 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.152941    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-lib-modules\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.153043 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.152995    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lm25\" (UniqueName: \"kubernetes.io/projected/025373f7-7969-47b6-9fe8-ee0b49d5bd74-kube-api-access-8lm25\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.153043 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.153028    5423 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-proc\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.153294 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.153050    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-sys\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.153294 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.153066    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-podres\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.153294 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.153120    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-proc\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.153294 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.153153    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/025373f7-7969-47b6-9fe8-ee0b49d5bd74-lib-modules\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.212801 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.212739    5423 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lm25\" (UniqueName: \"kubernetes.io/projected/025373f7-7969-47b6-9fe8-ee0b49d5bd74-kube-api-access-8lm25\") pod \"perf-node-gather-daemonset-7tpzl\" (UID: \"025373f7-7969-47b6-9fe8-ee0b49d5bd74\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.298168 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.298064    5423 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:41.682115 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:41.681968    5423 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"]
Nov 01 16:15:42.174105 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:42.174077    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-55crp_f521cd06-308b-48bb-a9a3-1ee4a82fbe04/kuadrant-console-plugin/0.log"
Nov 01 16:15:42.200497 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:42.200457    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl" event={"ID":"025373f7-7969-47b6-9fe8-ee0b49d5bd74","Type":"ContainerStarted","Data":"9e349e3b8abe10b1fc3dd095369fcb9d05d342929405f659ba044abb1385e29a"}
Nov 01 16:15:42.200691 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:42.200563    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:42.200691 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:42.200589    5423 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl" event={"ID":"025373f7-7969-47b6-9fe8-ee0b49d5bd74","Type":"ContainerStarted","Data":"81f0ec1ef45a635f552ed49750a4de49d397eaf1c3eda1862b3b3d62aa1efeff"}
Nov 01 16:15:42.385567 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:42.385525    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75bf7b5bf6-4vbfk_1c0d579d-61a9-4213-a72d-dcf2a340496b/manager/0.log"
Nov 01 16:15:43.002209 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:43.002178    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dkfnx_a4d27b75-03ea-410d-b323-86daa17409e9/limitador/0.log"
Nov 01 16:15:47.649225 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:47.649190    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9f778ddb6-jnqbk_0afcba4a-013c-42bd-8216-b9b91a2c4bde/manager/0.log"
Nov 01 16:15:48.216805 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:48.216778    5423 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl"
Nov 01 16:15:48.332286 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:48.332232    5423 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-7tpzl" podStartSLOduration=8.332218555 podStartE2EDuration="8.332218555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 16:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:42.316551349 +0000 UTC m=+7082.482036659" watchObservedRunningTime="2026-11-01 16:15:48.332218555 +0000 UTC m=+7088.497703853"
Nov 01 16:15:49.346576 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:49.346538    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75bf7b5bf6-4vbfk_1c0d579d-61a9-4213-a72d-dcf2a340496b/manager/0.log"
Nov 01 16:15:49.412194 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:49.412161    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jfqz9_607bc669-1dea-4cec-9a9b-5d726e2ea979/kube-state-metrics/0.log"
Nov 01 16:15:49.918029 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:49.917992    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jfqz9_607bc669-1dea-4cec-9a9b-5d726e2ea979/kube-rbac-proxy-main/0.log"
Nov 01 16:15:50.334096 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:50.334059    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jfqz9_607bc669-1dea-4cec-9a9b-5d726e2ea979/kube-rbac-proxy-self/0.log"
Nov 01 16:15:51.196448 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:51.196406    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-klphz_3585c408-fcee-40eb-9184-220f1f51a1ab/monitoring-plugin/0.log"
Nov 01 16:15:54.096526 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:54.096494    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvzfx_afc89564-0e45-485c-9c9f-07284715d978/node-exporter/0.log"
Nov 01 16:15:54.512161 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:54.512076    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvzfx_afc89564-0e45-485c-9c9f-07284715d978/kube-rbac-proxy/0.log"
Nov 01 16:15:54.943694 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:54.943667    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvzfx_afc89564-0e45-485c-9c9f-07284715d978/init-textfile/0.log"
Nov 01 16:15:55.364265 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:55.364235    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-k79t5_9a370ac2-b58d-42cf-91b8-2754d966dc04/kube-rbac-proxy-main/0.log"
Nov 01 16:15:55.787830 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:55.787747    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-k79t5_9a370ac2-b58d-42cf-91b8-2754d966dc04/kube-rbac-proxy-self/0.log"
Nov 01 16:15:56.205369 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:56.205338    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-k79t5_9a370ac2-b58d-42cf-91b8-2754d966dc04/openshift-state-metrics/0.log"
Nov 01 16:15:57.597695 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:57.597661    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qk6ww_3cb347b5-2346-4750-bf81-d005c9ae3fd9/discovery/0.log"
Nov 01 16:15:57.927048 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:57.926968    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fc9ccddb-v89s4_096fc6d0-c03c-4cb9-b330-3a71744e4470/kube-auth-proxy/0.log"
Nov 01 16:15:58.783417 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:58.783370    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-gkt8b_35bd3a7d-cb50-4c9e-b20c-c137bdbce8c7/istio-proxy/0.log"
Nov 01 16:15:59.446763 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:59.446719    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pw4lq_c3e03db8-a460-41e6-8eb4-a044121bd5dc/istio-proxy/0.log"
Nov 01 16:15:59.872257 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:15:59.872222    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wfr6h_d0dd2f89-8808-4b54-8255-e0c4dbd31fe9/istio-proxy/0.log"
Nov 01 16:16:00.514806 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:16:00.514777    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-688d45fc-hm5jh_ae261db3-a26e-4f0b-ad85-6a487a3faeda/payload-pre-processing/0.log"
Nov 01 16:16:00.727728 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:16:00.727696    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-dd858cb66-68dsg_2678e74b-8388-4f25-b832-934ccd98d338/payload-processing/0.log"
Nov 01 16:16:02.011357 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:16:02.011328    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb65fcfbb-txsmv_a61c1192-df73-4a89-88d3-9ebb9d890827/thanos-query/0.log"
Nov 01 16:16:02.427346 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:16:02.427318    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb65fcfbb-txsmv_a61c1192-df73-4a89-88d3-9ebb9d890827/kube-rbac-proxy-web/0.log"
Nov 01 16:16:02.844527 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:16:02.844502    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb65fcfbb-txsmv_a61c1192-df73-4a89-88d3-9ebb9d890827/kube-rbac-proxy/0.log"
Nov 01 16:16:03.288524 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:16:03.288448    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb65fcfbb-txsmv_a61c1192-df73-4a89-88d3-9ebb9d890827/prom-label-proxy/0.log"
Nov 01 16:16:03.706096 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:16:03.706064    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb65fcfbb-txsmv_a61c1192-df73-4a89-88d3-9ebb9d890827/kube-rbac-proxy-rules/0.log"
Nov 01 16:16:04.122128 ip-10-0-135-142 kubenswrapper[5423]: I1101 16:16:04.122099    5423 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bb65fcfbb-txsmv_a61c1192-df73-4a89-88d3-9ebb9d890827/kube-rbac-proxy-metrics/0.log"